I want to rip the contents of a pay website, but I have to log in to their web site on a web page to get access

Does anyone have any good tools for Windows for that?

I’m guessing that any such tools must have a built in browser, or be a browser plugin for it to work.

  • zabadoh@lemmy.mlOP
    2·
    1 year ago

    Webcopy looks promising if I can get the crawler part of it to work with this site’s authentication…

    edit: I couldn’t get Webcopy’s spider to authenticate correctly.

    Webcopy uses the deprecated version of Internet Explorer in Windows 10 as a module, and I can log into the website using the Capture Forms browser dialog, but the cookies or whatever else don’t translate over to the spider.