MnGCA

[King Boreas]

server too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busy

and over and over and over and over .....

[tomslusher]

I am having the same problem. I am trying to log a few KB caches I found today but I am not allowed in. I sent geocache.com an email telling them I will not renew my premium account if this problem is not solved. I assume all he needs is more server space or more bandwidth. Am I right?

tomslusher

[Moe the Sleaze]

Jeremy posted a few days ago that the servers have been getting hammered by 'bots'.
_________________
"Hi, I'm Moe, or as the women know me - Hey! You in the bushes."
-Moe, The Simpsons

[rickrich]

First, given that a person is seeking a certain dataset from the site, a bot will generate less total load than the same queries posed by the 'clicky-clicky-scroll-clicky-clicky' process. In the case of non-PQ queries, this is because the bots do not request any of the images on the site. In the case of PQ-queries, this is because with a bot there is no need to schedule daily PQ's at all, they can be easily performed only when you need them.

Second, bots are used because Jeremy has refused to provide features that people have repeatedly asked for. Among the things I can think of off the top of my head are:

2a) PQ's are limited to 500 caches, so you either have to manually generate multiple overlapping PQ's to get the entire metro area (increasing server load), or simulate GPX queries by scraping thru the web interface. This is exacerabated because Jeremy has provided only circular regions, not rectangular regions. Rectangular regions can be non-overlapping and do not require great-circle calculations. To accomplish the same with circular regions requires a 44% overlap and great circle calcs.

2b) PQ's are limited to 5 logs, which is often not enough. Once again, you are forced to use (either manually or with a bot) the web interface to see a sufficient number of logs.

2c) New cache notifications. If you want to be an FTF, you need timely new cache notifications. An email on Thursday does not cut the mustard. But AFAIK, gc.com does not offer instant email notofication when a new cache is placed. So people are forced to poll the site with a bot multiple times per day.

2d) Stats. Need I say more?

2e) Off-site cache listing and log backup.

2f) Search for all "latest logs" in an area so on a rainy day you can read up what your fellow cachers have been doing.

Third, Jeremy has made a number of changes in the last 9 months that make a bots job harder to do. The timing of these changes makes it pretty clear that he is aiming these changes at bots and not to provide new functionailty. Bots aren't going away until gc.com provides what people need from the site. So he has reaped what he has sown.

Fourth, Jeremy has not paid attention to the clicky-clicky usage pattern of humans accessing the site, and has not optimized the site for the access patterns that humans will generate. This generates additional and unneeded load, which is not caused by bots at all, that he could correct with a redesign of the site. Especially when in "logging caches mode", a large number of unnecessary database queries and page loads are generated.

Fifth, there was a fifth point, but I got brain block and can't remember it right now. I reserve the right to edit this post when my brain recovers.

Fifth (remembered now), the latest feature of modern web browsers is to pre-fetch pages in advance of the user requesting them. Mozilla does this, and I'm told that IE6 has something similar but not as good (I could be wrong about IE6). Its entirely possible that some of the increased load on the site is due to the growing popularity of this browser feature. Unfortunately, this browser feature looks and acts a lot like a bot, but without the precision targetting of just the data required that only a well-written bot can do.

Sixth, Jeremy is as guilty of leaching as the people he condemns. For example, when a new cache is placed, the map image comes from the US Census Tiger map server. New caches generate lots of website activity, and he is generating load on the Census bureaus server for his commercial purposes. The TMS people have requested that commercial sites do not do this. Hypocrit? You decide.

Jeremy wants to run the site "for profit". Unfortunately, this has forced him into a site design that does not scale very well other than by throwing hardware at the problem. He has made his bed and now he must lie in it. If the profits of the site are not sufficient to support the required new hardware investment every 3 months, then perhaps he should rethink his business. An open database would allow for a very scalable design, potentially harnessing the power of the computers of every geocacher.

So am I part of the problem? You decide.... I run a 'newest caches' bot every day at 5AM that scrapes 2 web pages. I run a 'stats' bot twice a week (Monday and Friday mornings). I run an FTF bot every hour that scrapes 1 web page. I don't use daily scheduled PQ's, rather, I run a bot-generated on-demand PQ 10 minutes before I leave the house. On average I run this about twice a week (some weeks never at alll, some weeks more than twice, depends on whether I am going out or not). I am a paying member, and I don't hunt MOCs or TBs. Every few months I use a bot to do an incremental backup of caches I have placed or found, including logs. This is always run during the week in the wee hours.

I'll probably end up getting banned from gc.com for posting this, but it had to be said.

-Rick

[s4xton]

RickRich - I'm with you 100%.

[15Tango]

In his gc.com forum posts on the same topic, it seems to me that Jeremy claims it's a bug in the system, and not related to the amount of users on the system. But, when I cache during the week, I'll go caching, then go to work at 3:30pm, and write my logs after I get home at 2:30am, and have no problems. Then when I cache on Saturday or Sunday, I'll write my logs that evening when everyone else does, and keep getting the Server to Busy thing. Last night, it took me over two hours to log 11 finds and 2 DNFs--good thing Saturday is a bad TV night and I didn't have any other plans. I'm also a Charter Member, why do I get chosen as the deadlock victim?
_________________
There comes a time in every young boy's life when he gets an irresistible urge to seek buried treasure.--Mark Twain

[rickrich]

Yes, I see that Jeremy has now backed off from blaming the 'bots' for his troubles.

He even started a new thread with essentially the same subject line as the old thread. No doubt he's hoping the old thread will fall off the first page and he can save some face.

Now, the data is that an "astounding" 50,000 logs were created during the last week, and this may be the source of his troubles. Even if everybody decided to log those 50,000 logs/week in just 4 hours on Saturday evening and 4 hours on Sunday evening that would amount to a transaction rate of less than 2 logs per second for the servers to handle. This really is not a large load.

I am glad to see Jeremy focusing now on the real source of the problem, which is the architecture of his system and the programming behind it. Unfortunately, he seems to be applying bandaid fixes (bailing water from a boat with a teaspoon). I expect he will get things under control for the time being, only to have this problem reoccur later this year. Its an architectural issue that he's going to have to face eventually. I saw he was looking for resumes

I also saw that robertlipe gently thru down the gauntlet for Jeremy to open up the source code so that more eyeballs could work on the problem. Nice chess move Robert! They will find Bin Laden before Jeremy opens his code, though.

If we go a period of time with no trouble and then the trouble surfaces again, I bet he will blame the 'bots' first. Anybody wanna take the bet?

-Rick

[robertlipe]

[rickrich]

Hi Robert! Nice to see you step into our forums. Team MnGCA, meet robertlipe, an alpha-geek from the great state of Tennessee. He's a virtual cache you should seek on your trip to MO this spring .

Robert is the team leader for gpsbabel, a fine piece of open source software thats at the heart of many page scrapers and many of the programs that feed your PDAs and GPSrs their waypoints.

Also, it looks like we will have a new leaderboard leader after Friday morning's scraping .

-Rick

[Marsha and Silent Bob]

[robertlipe]

Thanx for the welcome. If me joining the forum messes up your local stats site, I'll gladly unsubscribe or ask to be excluded or whatever. In TN we already have a lovely stats site so I don't need to skew your bell curves. (As an aside, you guys should get a piece of that action.) Rick and I have been friends and were coworkers for many years. I probably helped bait him into geocaching.

You're exactly right, Silent Bob. The domination of 'fan boys' in that crowd often gives it a ring of the Ministry of Truth. There are lots of things spoken and repeated as gospel that real techies know hold no water at all.

[Moe the Sleaze]

Anyone care for a little cheese with their...


_________________
"Hi, I'm Moe, or as the women know me - Hey! You in the bushes."
-Moe, The Simpsons

[s4xton]

[Marsha and Silent Bob]

[s4xton]