this post was submitted on 01 Jul 2023
15 points (85.7% liked)

Programming

17492 readers
43 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 1 year ago
MODERATORS
snowe@programming.dev
Ategon@programming.dev
MaungaHikoi
15
[HELP][Python] How to use Selenium correctly (discuss.tchncs.de)
submitted 1 year ago* (last edited 1 year ago) by PracticalParrot@discuss.tchncs.de to c/programming@programming.dev
22 comments fedilink hide all child comments
 

Hi there!
So I was hoping programming might be able to help with this. I am trying to learn how to use Selenium for personal projects.

I have tried using normal, straight up selenium but it does not support authorized proxies...
I have tried SeleniumBase, but when I got it working, there were WebRTC leaks that I could not avoid.
I have tried using Undetected-chrome but that too I was unable to get authorized proxies working...
For proxies, I tried seleniumwire as well. That seems to use its own SSL certificates which unfortunately as far as I can tell is easy to detect.
Main purpose for me is web scraping and or simple login to X site, enter X info and close.

So my question is... What is the proper way to use Selenium for automating functions while avoiding detection?

Thank you in advance.

EDIT: I made this post because I am hitting a wall at every turn, and I feel like I might just be approaching this the wrong way, or maybe I'm just missing something crucial. I was hoping someone with more experience could explain to me what is the correct way of doing it without cobbling it together with duct tape.

you are viewing a single comment's thread
view the rest of the comments
[–] ake@lemmy.dbzer0.com 2 points 1 year ago (4 children)

have you considered using something else? like playwright/puppeteer?

[–] PracticalParrot@discuss.tchncs.de 2 points 1 year ago (3 children)

I have considered it but since I only know Python I have yet to try it. Might be time to learn javascript :)

[–] ake@lemmy.dbzer0.com 6 points 1 year ago (2 children)

https://playwright.dev/python/docs/intro

[–] PracticalParrot@discuss.tchncs.de 2 points 1 year ago

Wow I had no idea! Thanks for sharing I'll look into it.

[–] PracticalParrot@discuss.tchncs.de 1 points 1 year ago

After getting it working with proxies I realize they have the exact same webRTC issue. I assume it's for the same reason as Selenium having it. Unfortunate but there you have it.