Check if account is shadowbanned on the run by jakubkrysl · Pull Request #2059 · RocketMap/RocketMap

jakubkrysl · 2017-05-23T16:52:30Z

This PR aims to identify if accounts are shadowbanned / get shadowban on the run.

Description

One can identify shadowbanned accounts by missing pokemons in their scan. So if current account sees only "common" pokemons and no "rares", it pulls nearby active pokemons from database and compares them to what the current account sees. If there are any rare pokemons around and the current account can't see them, it sends a warning with pokemon IDs saying, the account is shadowbanned. It also writes the account to "acc_shadowbanned.csv" (only if not there yet) and sends it to recycler for -ari + 24h.
It also counts consecutive non-rare scans and if the number exceeds 20, it sends the account to recycle for -ari. It does not write to "acc_shadowbanned.csv" in this case, because this is not 100% reliable, as some rural areas can have only common pokemons. The number of consecutive counts is so high on purpose, because we have to give the previous method some time to work. This way fresh accounts get into rotation and we can use the 1st method.

Motivation and Context

Who would not want to know, if their account gets blinded? And sort them out...

How Has This Been Tested?

I was running -st 20 with blinded accounts few hours to identify common pokemons. Than I took few of these accounts and let them run the same area with not yet blinded accounts. Here is a log of found blinded account:

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.

blueberry0101 · 2017-05-23T17:41:12Z

Yes, there is a better way.

common = [16, 19, 27, 29, 32, 41, 43, 46, 52, 54, 60, 69, 72, 74, 98, 118, 120, 129, 161, 165, 167, 177, 183, 187, 194, 198, 209, 218] rare_ids = [x for x in range(1, 250) if x not in common]

I meant better way finding common pokemons than using this kind of list of IDs.Tthis is mostly cosmetic change, but looks better, you are right :)

range(1,252)

@aRengo the rare_ids list is constructed from range, but IDs of common pokemons are substracted from the list. So the range is not complete, that's the whole point :D

jakubkrysl · 2017-05-24T09:55:11Z

I just added the rest of confirmed common pokemons, source reddit.
I am looking into possiblity of confirming shadowbanned accounts by checking the store for items, it seems shadowbanned accounts can't see any items buyable by coins, they only see real money exchanges. To clarify: blindness is a symptom of shadowban.
So far I traced the request to GET_STORE_ITEMS protobuf object, but I have no idea how to access it as it is in platform of pgoapi and not with the rest of protobufs. Could definitely use some help there...

EDIT: Checking store for missing items does not seem to be a reliable way to check for shadowban. The account has to be flagged too for this to happen, but flag != shadowban. So no point in doing another check this way, if it can't check for shadowban alone...

sebastienvercammen · 2017-05-24T19:13:21Z

@Thetacz

2: Preferably not. The idea of using the shop to identify blind accounts (because they could only see coins) was a good one, but it's apparently not true for everyone. As an alternative, I suggest looking at scan data: it seems blind accounts don't get all details the regular accounts are getting, as the 'enabled' field was missing and triggered a KeyError. 'enabled' is a field that's usually always received.

But we've also started working on a PGoClient wrapper around PGoAPI, which could include a status such as banned/flagged/blind, and which could automatically update the account status after every request. @aRengo

3: Look at the flow for banned accounts. Iirc, we remove them from queue, output to logs, and never retry that account.

jakubkrysl · 2017-05-24T21:29:12Z

Now it properly recycles the accounts (in case the shadowbans get lifted after some time - stil not confirmed). I can either leave it as it is and those accounts will get re-tried after user specified time, set some fixed time for shadowbanned (needs more reaserch, being done at #pogodev) or completely discard the account (or leave it there and add if that will skip shadowbanned accounts)

jakubkrysl · 2017-05-24T21:52:18Z

@sebastienvercammen Just compared response from shadowbanned and not. Those responses only differ in visible pokemons (aside from account specific - types of pokemon it seen, stats etc...). They both have the same amount of 'enabled' fields (same location). 'enabled' field only occurs in forts, could not see it anywhere else. All 'enabled' fields were set to true in both responses.

EDIT: missclick on close, sry :/

Alderon86 · 2017-05-25T12:44:10Z

Please use the logger (%s) instead of string with {} some stuff {} in it.format(foo,bar)

@Alderon86 I used %s before, but changed it for consistency with line #836. If you insist so much, I will change it in a heatbeat. :)

Or should I split the mesage to more lines? (Similar way as is done in models.py when logging empty spawnpoint - line 2375)

Alderon86 · 2017-05-25T13:35:06Z

status['message'] = ( 'Account %s could not find pokemon ID: %s. ' + 'Possibly shadowbanned, switching accounts... ' ,e.account['username'], e.missed_ids)

Alderon86 · 2017-05-25T13:35:54Z

status['message'] = ( 'Account %s could not find pokemon ID: %s. ' + 'Possibly shadowbanned, switching accounts... ', e.account['username'], e.missed_ids)

https://stackoverflow.com/a/11589591
But you are right, looks nicer.

Alderon86 · 2017-05-25T17:50:02Z

1.list of rare pokemons constains some common ones that I have not encountered yet in my area. For example: Have not seen bulba in ages, any nest to test if he is common? 2.Do we want to identify blinded accounts based on their inability to find rare pokemons? (Warn after X consecutive rareless finds...) 3.What do we want to do with identified blinded accounts?

1.Bulba and any other of the starter pokemons are not common.
2.I like the idea of giving a warning after 5 times fails to scan a rare, after 10 fails mark it as shadowbanned
3.remove from rotation and give a log/status warning that the account is blind/shadowbanned and is removed from rotation.

Ofc just my two cents

Borges-Wellington · 2017-06-01T20:55:36Z

Does anyone know how to implement mapbox leaflet in place of gmap? For Internet browsing and others. I have other instances that do a scan and wanted an interface booklet

Buncertainc · 2017-06-02T16:50:41Z

It seems that No.77 Ponyta should also be included in the common_ids.

jakubkrysl · 2017-06-03T09:20:46Z

@Buncertainc Do you have some source for that? Or any location of Ponyta nest, so I can test it?

Buncertainc · 2017-06-03T14:12:01Z

@Thetacz No, Sorry. It's only from my accounts' experience. I have several blinded accounts which are able to find Ponyta.

tomballgithub · 2017-06-06T03:28:19Z

@Thetacz Ponyta nest @ 30.168453, -95.7407196

sebastienvercammen · 2017-06-06T10:49:27Z

@wellington1092 This is not the place for that.

@Thetacz Awaiting changes, or ready for review?

jakubkrysl · 2017-06-06T18:29:39Z

@Buncertainc Thanks for the tip, it seems to be true.
@tomballgithub Thanks, using that nest I confirmed, ponyta is among common pokemons. Adding it to the list.
@sebastienvercammen Just added ponyta to common list, so it is ready for review. Sorry for the delay, too much new stuff going on IRL :)

pki0 · 2017-06-09T21:35:38Z

What about a new column "blind" in workerstatus? If an account hits the shadowban set blind to '1', else '0'.
This would make scripts for automatic restart/account changing more easy and comfortable.
Just a suggestion .

Edit: Or a column named "Status" with OK/Blind/Banned as options

jakubkrysl · 2017-06-14T17:42:44Z

@pokemonkiel That seems like out of scope of this PR as that should be solved by proper account manager. Generally I like the idea, but accounts get unshadowbanned, so there would have to be a way to check and reset the flag in DB. And that is definitely out of scope and work for proper account manager...Now we just send accounts to rest.
I implemented saving to file, so there is room for custom management. Without the need for future massive code change on RM side.

Kimuji · 2017-06-17T12:49:16Z

I'm currently trying your PR.
From what i saw it seems that event related pokemons are still displayed for shadowbanned accounts.
So it means during events this PR isn't detecting those blind accounts.

I understand the logic behind your PR, but why is the warning "spawnpoint xxx has no Pokemon N times in a row." not used in a way or an other in the detection ?

SiteEffect · 2017-06-18T19:26:38Z

An option would be to make the list a flag with current list as default and let map maintainers adjust them during an event.

SiteEffect · 2017-06-19T16:55:38Z

This is the current list common_list for the live event:

    common_ids = [16, 19, 23, 27, 29, 32, 37, 41, 43, 46, 52, 54, 58, 60,
                  69, 72, 74, 77, 81, 90, 98, 118, 120, 129, 155, 161, 165,
                  167, 177, 183, 187, 191, 194, 198, 209, 218, 220, 228]

If you find anything else being common, let us know. I am not sure about Lapras # 131.

friscoMad · 2017-06-20T07:39:53Z

I am not sure about this system I do see what -redacted- does a lot simpler just check wild + nearby for rare pokemons, nearby list is large and is uncommon to don't have any rare pokemon in the list.
This system seem more exact except when the whole map start blinding at the same time, I would see this being better if we do pause blind accounts asap

SiteEffect · 2017-07-07T08:03:03Z

Re-check list and rebase, please :-)
Looks like the shadowbans are back.

SiteEffect · 2017-07-15T15:01:57Z

There is a conflict with models.py left. You need to rebase to the new API objects instead of dicts.

jakubkrysl · 2017-07-16T20:42:58Z

Sadly I don't have any free time now. Soonest date I can get to this is 4 days from now, but more probable is next week. Sorry guys for keeping you waiting with broken code.

SiteEffect · 2017-07-17T23:33:25Z

No hurry 👍 enjoy!

It's ALIVE!!!!!

jakubkrysl · 2017-07-29T13:28:53Z

Missclick on wrong branch, reopening and working again :)

Happy Travis is happy

tobias86aa · 2017-07-29T13:38:16Z

Nice :)

boby361 · 2017-07-31T02:48:10Z

So happy to see this. Have been so patient.

boby361 · 2017-08-04T06:37:29Z

So i have a weird bug. I restarted my server and i got this error:
Traceback (most recent call last):
File "runserver.py", line 20, in
from pogom.app import Pogom
File "C:\Servers\RocketMap-Boby361\pogom\app.py", line 19, in
from .models import (Pokemon, Gym, Pokestop, ScannedLocation,
File "C:\Servers\RocketMap-Boby361\pogom\models.py", line 35, in
from .customLog import printPokemon
File "C:\Servers\RocketMap-Boby361\pogom\customLog.py", line 5, in
args = get_args()
File "C:\Servers\RocketMap-Boby361\pogom\utils.py", line 41, in wrapper
rv = function(*args)
File "C:\Servers\RocketMap-Boby361\pogom\utils.py", line 679, in get_args
args.username.pop(i)
IndexError: pop index out of range

If i rename my shadow ban file to somethhing else. it works. The file hasn't been touched in awhile. The formatting seems all proper (UTF-8). there is nothing missing. no extra space. nothing. it fixes itself if i remove accounts out. It happened with 17 accounts in it. Thoughts? Edit Its happened with less accounts now.

jakubkrysl · 2017-08-06T08:00:29Z

@boby361 This is still PR so things change from time to time and some are not backwards compatible. I added 4th field to the file, timestamp. Your old file has probably only 3 fields. So either check the code and add proper timestamp or delete it and let it create a new one.
Also check if you have the same account more than once in the file. Could not check the possibility, but would be very interested if it happens.

boby361 · 2017-08-06T21:52:26Z

Yeah the file only contains lines with the 4 fields. Everything seems to work properly. It's just weird how it randomly throws this issue. I have other people having the same issue with the same fix (using the same fork. Although it's just a up to date RM develop branch + this PR) it happened to me on windows server 2016 although one of my scanning contributors had same issue on Ubuntu.

A subtle blend of sLoPPy's detection and PR RocketMap#2059 with a bunch of tweaks of my own. Looks for "non-common" Pokemon on each scan to update non-rare scan counts accordingly. No account switching, just logs. TODO: Put this stuff in a database

jorgitoh · 2017-10-05T13:14:03Z

Hi. Any progress with this PR?. Now we have Temp Ban and shadow bans again. I think is time to put an eye in this PR.

sebastienvercammen · 2017-10-05T18:58:55Z

@jorgitoh Then go ahead and finish it yourself, it's open source. Otherwise stay silent if you have nothing to contribute.

A subtle blend of sLoPPy's detection and PR RocketMap#2059 with a bunch of tweaks of my own. Looks for "non-common" Pokemon on each scan to update non-rare scan counts accordingly. No account switching, just logs. TODO: Put this stuff in a database

blueberry0101 reviewed May 23, 2017

View reviewed changes

jakubkrysl closed this May 24, 2017

jakubkrysl reopened this May 24, 2017

Alderon86 suggested changes May 25, 2017

View reviewed changes

Alderon86 reviewed May 25, 2017

View reviewed changes

Alderon86 suggested changes May 25, 2017

View reviewed changes

jakubkrysl force-pushed the check-blinded branch from 9224de9 to 9ae468b Compare May 30, 2017 12:37

jakubkrysl changed the title ~~[WIP] Check if account is blinded on the run~~ Check if account is shadowbanned on the run May 30, 2017

sebastienvercammen added this to the 4.0.0 milestone Jun 1, 2017

sebastienvercammen added awaiting review enhancement labels Jun 1, 2017

05b4e01

It's ALIVE!!!!!

jakubkrysl closed this Jul 29, 2017

jakubkrysl force-pushed the check-blinded branch from b30b384 to 44baabd Compare July 29, 2017 13:26

jakubkrysl reopened this Jul 29, 2017

6c6f19b

Happy Travis is happy

jakubkrysl added 3 commits July 30, 2017 11:04

Discard shadowbanned account while loading

be649ba

Prevent false positives

d1c91b5

Fix ile transfer and Travis

1a46d8a

Alderon86 added a commit to Alderon86/RocketMap that referenced this pull request Jan 4, 2018

Update in shadowban logic according to RocketMap#2059

40e859a

Alderon86 added a commit to Alderon86/RocketMap that referenced this pull request Jan 6, 2018

Update in shadowban logic according to RocketMap#2059

bc5e853

Alderon86 added a commit to Alderon86/RocketMap that referenced this pull request Jan 7, 2018

Update in shadowban logic according to RocketMap#2059

2e32ee9

Uh oh!

Conversation

jakubkrysl commented May 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

blueberry0101 May 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakubkrysl commented May 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sebastienvercammen commented May 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakubkrysl commented May 24, 2017

Uh oh!

jakubkrysl commented May 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakubkrysl May 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Alderon86 commented May 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Borges-Wellington commented Jun 1, 2017

Uh oh!

Buncertainc commented Jun 2, 2017

Uh oh!

jakubkrysl commented Jun 3, 2017

Uh oh!

Buncertainc commented Jun 3, 2017

Uh oh!

tomballgithub commented Jun 6, 2017

Uh oh!

sebastienvercammen commented Jun 6, 2017

Uh oh!

jakubkrysl commented Jun 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pki0 commented Jun 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakubkrysl commented Jun 14, 2017

Uh oh!

Kimuji commented Jun 17, 2017

Uh oh!

SiteEffect commented Jun 18, 2017

Uh oh!

SiteEffect commented Jun 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

friscoMad commented Jun 20, 2017 • edited by sebastienvercammen Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SiteEffect commented Jul 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

jakubkrysl commented May 23, 2017 •

edited

Loading

blueberry0101 May 23, 2017 •

edited

Loading

jakubkrysl commented May 24, 2017 •

edited

Loading

sebastienvercammen commented May 24, 2017 •

edited

Loading

jakubkrysl commented May 24, 2017 •

edited

Loading

jakubkrysl May 25, 2017 •

edited

Loading

Alderon86 commented May 25, 2017 •

edited

Loading

jakubkrysl commented Jun 6, 2017 •

edited

Loading

pki0 commented Jun 9, 2017 •

edited

Loading

SiteEffect commented Jun 19, 2017 •

edited

Loading

friscoMad commented Jun 20, 2017 •

edited by sebastienvercammen

Loading

SiteEffect commented Jul 7, 2017 •

edited

Loading

SiteEffect commented Jul 15, 2017 •

edited

Loading

boby361 commented Aug 4, 2017 •

edited

Loading

jakubkrysl commented Aug 6, 2017 •

edited

Loading