Douyin video collection issues

[dale-wahl]

Douyin provides lots of extra videos that are picked up by Zeeschuimer and telling the difference can be difficult.

Recommend page (also douyin.com homepage)

Collecting ~36 videos on load; 8 of which will be shown in scroll
Two additional video groups:

• Front Page videos - sometimes a package containing those videos (detectable by Zeeschuimer as unique) are also loaded
• These videos are not reloaded if the user then navigates to the Front Page meaning we would need to cache these and return them if the user then navigates to the Front page... possibly in X amount of time?
• Other videos for unknown reasons are also loaded that are in the same structure as normal videos
  • chime_video_list possible indicator; may refer to "trending videos", but I have not located where these are displayed yet

Front page

Collecting ~80 videos on load. Was missing some embedded videos (moved in data structure); fixed in ef16ea7

[dale-wahl]

As of commit b64df2e

• Home page (douyin.com aka Recommend (推荐)) now only collects videos that are actually loaded in the scroll. This means skipping videos that may be on other pages (such as those for the Front Page (首页)) or appear when selecting videos (e.g., recommended videos). Forcing a refresh should then collect those videos on those videos on those pages.

• Front page (首页) (https://www.douyin.com/discover) verified to be collecting correctly. This page most likely to have cached videos from other pages and thus requires a refresh.

• User pages work

• Search works and collects all the video cards.

  • Some searches appear to have one large video at the top. This is rare and I cannot seem to recreate it. But I do not believe it was captured that one time I saw it...

• https://www.douyin.com/hot looks good

• Channel pages look good

• Individual video pages issues remain:

  • First video not being collected
  • Collecting extra videos: I cannot tell the difference between the objects yet only some appear to be in the list/queue.