Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth #204

Open
scuc opened this issue Jun 29, 2023 · 19 comments · May be fixed by #205
Open

403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth #204

scuc opened this issue Jun 29, 2023 · 19 comments · May be fixed by #205

Comments

@scuc
Copy link

scuc commented Jun 29, 2023 •
edited
Loading

Is anyone else seeing this 403 Client Error?
I started to see it on May 09, my script runs automated
so I didn't actually notice this issue in my logs
until last week.
I thought maybe it was an MFA issue,
so I've tried to add MFA as explained in the wiki,
but I cant even get past the 403 error.

What version of Python are you using (python -V)?

Python 3.10.4

What operating system and processor architecture are you using (python -c 'import platform; print(platform.uname());')?

('Darwin', 'MacPro-Mojave.local', '18.7.0', 'Darwin Kernel Version 18.7.0: Tue Jun 22 19:37:08 PDT 2021; root:xnu-4903.278.70~1/RELEASE_X86_64', 'x86_64', 'i386')

Which Python packages do you have installed (run the pip freeze or pip3 freeze command and paste output)?

arlo==1.2.59 (also test  with .64 version)
cachetools==5.3.1
certifi==2023.5.7
cffi==1.15.0
charset-normalizer==3.1.0
click==8.1.3
cloudscraper==1.2.60
cryptography==37.0.2
google-api-core==2.11.1
google-api-python-client==2.90.0
google-auth==2.21.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==1.0.0
googleapis-common-protos==1.59.1
httplib2==0.22.0
idna==3.4
monotonic==1.6
oauthlib==3.2.2
paho-mqtt==1.6.1
pickle-mixin==1.0.2
protobuf==4.23.3
pyaarlo @ git+https://github.com/twrecked/pyaarlo@2d6941dc903fe2fca01ac9d82fe708d1299ebe81
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pycryptodome==3.14.1
pyparsing==3.1.0
PySocks==1.7.1
PyYAML==6.0
requests==2.31.0
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
rsa==4.9
six==1.16.0
sseclient==0.0.22
Unidecode==1.3.4
uritemplate==4.1.1
urllib3==1.24

Which version of ffmpeg are you using (ffmpeg -version)?

ffmpeg version 1.2-tessus Copyright (c) 2000-2013 the FFmpeg developers
  built on Mar 15 2013 01:18:55 with llvm-gcc 4.2.1 (LLVM build 2336.1.00)
  configuration: --prefix=/Users/tessus/data/ext/ffmpeg/sw --as=yasm --extra-version=tessus --disable-shared --enable-static --disable-ffplay --enable-gpl --enable-pthreads --enable-postproc --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-libspeex --enable-bzlib --enable-zlib --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libxavs --enable-version3 --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvpx --enable-libgsm --enable-libopus --enable-fontconfig --enable-libfreetype --enable-libass --enable-filters --enable-runtime-cpudetect
  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100

Which Arlo hardware do you have (camera types - [Arlo, Pro, Q, etc.], basestation model, etc.)?

not relevant

What did you do?

This code was working for me up without MFA until May 09 2023 - I thought maybe it was an MFA issue, so
I added MFA to the code, but still getting the same errrors.

from arlo import Arlo
from datetime import timedelta, date, datetime
import logging
# from pathlib import Path

import config
import check_subdirs as chksub

config = config.get_config()

USERNAME = config['creds']['username']
PASSWORD = config['creds']['password']
MFA = config['creds']['mfa']

logger = logging.getLogger(__name__)

ROOTPATH = config['paths']['root_path']

def download_mp4s(): 
    """
    Iterate over the videos in the arlo S3 bucket, compare against videos stored locally, dowload 
    all new files, rename downloaded files with a datetime, skip those that have already been downloaded. 
    """
    # Instantiating the Arlo object automatically calls Login(), which returns an oAuth token that gets cached.
    # Subsequent successful calls to login will update the oAuth token.
    try: 
        arlo = Arlo(USERNAME, PASSWORD, MFA)
        logger.info("Successfully logged into Arlo account")

        today = (date.today()-timedelta(days=0)).strftime("%Y%m%d")
        start_date = (date.today()-timedelta(days=30)).strftime("%Y%m%d")

        # Get all of the recordings for a date range.
        library = arlo.GetLibrary(start_date, today)
        logger.info("Connected to the Arlo Library")
        devices = arlo.GetDevices()
        # print(devices)

        device_list = []

        for device in devices:
            deviceId = device["deviceId"]
            deviceName = device["deviceName"]
            device_dict = {device["deviceId"]: device["deviceName"]}
            device_list.append(device_dict)

        logger.info(f"Devices found in library: {device_list}")

        # Iterate through the recordings in the library.
        download_count = 0
        for recording in library:

            deviceId = recording["deviceId"]

            deviceNum = next(i for i,d in enumerate(device_list) if deviceId in d)

            deviceName = device_list[deviceNum][deviceId]

            videofilename = datetime.fromtimestamp(int(recording['name'])//1000).strftime('%Y-%m-%d_%H-%M-%S') + '_' + recording['uniqueId'] + '.mp4'

            ###################
            # The videos produced by Arlo are pretty small, even in their longest, best quality settings,
            # but you should probably prefer the chunked stream (see below).
            ##################
            #    # Download the whole video into memory as a single chunk.
            #    video = arlo.GetRecording(recording['presignedContentUrl'])
            #	 with open('videos/'+videofilename, 'wb') as f:
            #        f.write(video)
            #        f.close()
            ################33

            root_dir = (ROOTPATH + deviceName + '/')
            mp4_path = chksub.check_subdirs(root_dir, deviceName, videofilename)

            if mp4_path.is_file() is not True:
                stream = arlo.StreamRecording(recording['presignedContentUrl'])
                with open(str(mp4_path), 'wb') as f:
                    for chunk in stream: 
                        f.write(chunk)
                    f.close()
                    download_msg = 'Downloaded '+ videofilename+' from '+ recording['createdDate']+'.'
                    dowload_path_msg = "Download path: " + str(mp4_path)
                    download_count += 1
                    logger.info(download_msg)
                    logger.info(dowload_path_msg)
            else:
                # Get video as a chunked stream; this function returns a generator.
                skip_msg = "Skipping: " + str(videofilename)
                logger.debug(skip_msg)
                continue

        # Delete all of the videos you just downloaded from the Arlo library.
        # Notice that you can pass the "library" object we got back from the GetLibrary() call.
        # result = arlo.BatchDeleteRecordings(library)

        if download_count != 0: 
            logger.info(f"Total videos downloaded: {download_count}")
        else: 
            logger.info(f"No new videos for download.")
        return

    except Exception as e:
        logger.exception(e)
      

if __name__ == '__main__':
    download_mp4s()

What did you expect to see?

I use the code listed above to download a local copy of my arlo videos.

What did you see instead?

    ================================================================
                ARLO Download Script - Start
                    Wednesday, 28. June 2023 09:46PM
    ================================================================
    
2023-06-28 21:46:34,815 | ERROR | Function: download_mp4s() | Line 103 | 403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth
Traceback (most recent call last):
  File "/Users/myname/_Github/ARLO-S3-Download/arlo_download.py", line 27, in download_mp4s
    arlo = Arlo(USERNAME, PASSWORD, MFA)
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/arlo.py", line 69, in __init__
    self.LoginMFA(username, password, google_credential_file)
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/arlo.py", line 201, in LoginMFA
    auth_body = self.request.post(
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/request.py", line 80, in post
    return self._request(url, 'POST', params=params, headers=headers, raw=raw)
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/request.py", line 56, in _request
    r.raise_for_status()
  File "/Users/myname/_Github/ARLO-S3-Download/.venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth
2023-06-28 21:46:34,820 | INFO | Function: main() | Line 89 | 
        ================================================================
                    ARLO Download - Complete
                        Wednesday, 28. June 2023 09:46PM
        ================================================================

Does this issue reproduce with the latest release?

yes. same results. in .59 and .64,
ive tried connecting through a VPN to see if I my IP was being blocked,
i can also confirm that I have tried multiple user accounts - in each case
I can log in just fine through the arlo web portal, but get the 403 when using the Arlo code.
@scuc
Copy link
Author

scuc commented Jun 30, 2023

FYI, it looks like scrypted issued a patch to deal with the exact same type of 403 issue. Also, the issue appeared around May 10 right around same time as the issue that I am seeing.

koush/scrypted@0a020f2

Trying to understand exactly what they did to fix the issue, no comments in the code.

@misterek
Copy link

misterek commented Jul 2, 2023

Just took a quick look, but it looks to me like the assumption is that CloudFlare is causing the issue. So they hardcoded some IP's direct to Arlo (in base64), and if the main one fails, they try to find one of those with the correct SSL cert, and use that instead.

Doesn't seem like a great option, since those IP's can change at any time (and if I checked right, at lest one is already not pointing at Arlo.)

@tomelsj
Copy link

tomelsj commented Jul 4, 2023

Hmm, not even sure the scrypted solution works at all anymore. The API is protected behind CloudFront. Have not found any way to access it. And all the IP:s are broken as well.

Perhaps better luck building a "private server" for the Arlo cameras.

@misterek
Copy link

misterek commented Jul 4, 2023

I'd guess CloudFlare is doing some sort of TLS fingerprinting or similar. I played around a bit, both using Cloudscraper and swapping around the TLS cipher order. Didn't have any luck.

I'll wait a bit to see what anyone else comes up with, but at this point I may just go with another vendor if Arlo doesn't want me to have access to my own recordings.

@misterek
Copy link

misterek commented Jul 4, 2023

Well, given this: koush/scrypted@8eb533c
Maybe they've found a way around using cloud scraper. Note that they had to update their IP list as well. I expect that'll have to be done regularly.

@m0urs
Copy link
Contributor

m0urs commented Jul 7, 2023 •
edited
Loading

I had a similar issue with Arlo and Cloudscraper based on another Python module (https://github.com/m0urs/arlo-fhem). For me, the Cloudflare login issue has been resolved by adding this parameter to the cloudscraper call:

self._session = cloudscraper.create_scraper(ecdhCurve='secp384r1')

together with Cloudscraper 1.2.71 and the following User Agent String:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58

Not sure if the last two things are related to that, but the Cloudscraper parameter definitely has been necessary for me. Maybe it could resolve the issue for others as well ...

@scuc
Copy link
Author

scuc commented Jul 15, 2023

@m0urs thank you for the suggestions, but I'm unclear as to where exactly i should add the cloud scraper call into my code.
A search of the arlo library has no ref to cloudscraper.

@m0urs
Copy link
Contributor

m0urs commented Jul 15, 2023

To be honest, I do not know this code here, but I guess that something similar is used to handle the Cloudflare checking for HTTP requests. Maybe @jeffreydwalter should know how this code is doing it and maybe there are similar settings.

@tomelsj
Copy link

tomelsj commented Jul 17, 2023

In arlo.py at line 139, something like this could be done:
self.request = cloudscraper.create_scraper(ecdhCurve='secp384r1')
Instead of Request()
Need to import cloudscraper too like import cloudscraper along with other imports. I cannot verify this actually works at this moment.

@scuc
Copy link
Author

scuc commented Jul 20, 2023

@tomelsj thanks for the suggestions, i tried it, unfortunately, still get a 403 error, so frustrating -
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth

@m0urs
Copy link
Contributor

m0urs commented Jul 20, 2023 •
edited
Loading

Did you wait a day between the last login try without the parameter and the next try with the parameter? If not, please wait some hours until Cloudflare has unbanned your IP address.

@scuc
Copy link
Author

scuc commented Jul 21, 2023

@m0urs yes, i updated my code and waited a few days before trying again
unfortunately I still see a 403-Forbidden response.

@scuc
Copy link
Author

scuc commented Aug 30, 2023

I still have not been able to get this to work.
is there someone who actually has the login working with Arlo library?
if so, would you be able to send a PR for this package to addresses this issue?

@Joreid
Copy link

Joreid commented Sep 7, 2023

Any updates for this error?

@ahass-thedev
Copy link

Hoping for any leads/fixes for this error. Unsure if it is possible to proceed without a fix

@bjia56 bjia56 linked a pull request Sep 16, 2023 that will close this issue
Open
@scuc
Copy link
Author

scuc commented Oct 13, 2023 •
edited
Loading

@bjia56 - im circling back to this issue - trying out your fix, but getting the error below.
not sure what's wrong here - have my gmail credential expired?

2023-10-13 13:21:04,566 | INFO | Function: autodetect() | Line 49 | file_cache is only supported with oauth2client<4.0.0
2023-10-13 13:21:09,778 | ERROR | Function: download_mp4s() | Line 103 | ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
Traceback (most recent call last):
 File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo_download.py", line 27, in download_mp4s
   arlo = Arlo(USERNAME, PASSWORD, google_credential_file=MFA)
 File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo.py", line 69, in __init__
   self.LoginMFA(username, password, google_credential_file)
 File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo.py", line 256, in LoginMFA
   ).execute()
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
   return wrapped(*args, **kwargs)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 923, in execute
   resp, content = _retry_request(
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 191, in _retry_request
   resp, content = http.request(uri, method, *args, **kwargs)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google_auth_httplib2.py", line 209, in request
   self.credentials.before_request(self._request, method, uri, request_headers)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/auth/credentials.py", line 134, in before_request
   self.refresh(request)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/credentials.py", line 319, in refresh
   ) = reauth.refresh_grant(
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/reauth.py", line 349, in refresh_grant
   _client._handle_error_response(response_data, retryable_error)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/_client.py", line 69, in _handle_error_response
   raise exceptions.RefreshError(
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
2023-10-13 13:21:09,783 | INFO | Function: main() | Line 89 |

@scuc
Copy link
Author

scuc commented Oct 13, 2023

@bjia56 - im circling back to this issue - trying out your fix, but getting the error below. not sure what's wrong here - have my gmail credential expired?

2023-10-13 13:21:04,566 | INFO | Function: autodetect() | Line 49 | file_cache is only supported with oauth2client<4.0.0
2023-10-13 13:21:09,778 | ERROR | Function: download_mp4s() | Line 103 | ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
Traceback (most recent call last):
 File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo_download.py", line 27, in download_mp4s
   arlo = Arlo(USERNAME, PASSWORD, google_credential_file=MFA)
 File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo.py", line 69, in __init__
   self.LoginMFA(username, password, google_credential_file)
 File "/Users/username/Documents/Github/arlo-cloudflarefix/arlo.py", line 256, in LoginMFA
   ).execute()
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
   return wrapped(*args, **kwargs)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 923, in execute
   resp, content = _retry_request(
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 191, in _retry_request
   resp, content = http.request(uri, method, *args, **kwargs)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google_auth_httplib2.py", line 209, in request
   self.credentials.before_request(self._request, method, uri, request_headers)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/auth/credentials.py", line 134, in before_request
   self.refresh(request)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/credentials.py", line 319, in refresh
   ) = reauth.refresh_grant(
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/reauth.py", line 349, in refresh_grant
   _client._handle_error_response(response_data, retryable_error)
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/google/oauth2/_client.py", line 69, in _handle_error_response
   raise exceptions.RefreshError(
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
2023-10-13 13:21:09,783 | INFO | Function: main() | Line 89 |

ok never mind, I just went through the google MFA steps again and re-created the credentials and now it works!!!!
wow, your PR should def get merged into the master! thank you so much for the fix!

@cgmckeever
Copy link

Last I knew @jeffreydwalter was no longer using this library .. maybe time for a hand-off of fork?

@scuc
Copy link
Author

scuc commented Oct 14, 2023

Last I knew @jeffreydwalter was no longer using this library .. maybe time for a hand-off of fork?

yeah, I hope we can keep this library alive, I have a lot of money invested in Arlo cams. And all I really
want to do is download my videos for a local archive, so in that regard, the arlo package has been been a huge help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix cloudflare 403 with curl_cffi bjia56/arlo
7 participants
@cgmckeever @m0urs @scuc @misterek @ahass-thedev @tomelsj @Joreid