MnGCA Home MnGCA
Minnesota Geocaching Association
 
 FAQFAQ   SearchSearch   User listUser list   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Server too busy
Goto page 1, 2, 3, 4, 5, 6  Next
 
Post new topic   Reply to topic    MnGCA Forum Index -> Geocaching Websites
View previous topic :: View next topic  
Author Message
King Boreas
Geocacher


Joined: 16 Dec 2002

Posts: 2442

PostPosted: Sat Feb 28, 2004 10:49 pm    Post subject: Server too busy Reply with quote

server too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busyserver too busy server too busy

and over and over and over and over .....
Back to top
View user's profile Send private message Send e-mail
tomslusher
Geocacher


Joined: 02 Jan 2003

Posts: 182

PostPosted: Sat Feb 28, 2004 11:00 pm    Post subject: Reply with quote

I am having the same problem. I am trying to log a few KB caches I found today but I am not allowed in. I sent geocache.com an email telling them I will not renew my premium account if this problem is not solved. I assume all he needs is more server space or more bandwidth. Am I right?

tomslusher
Back to top
View user's profile Send private message
Moe the Sleaze
Geocacher


Joined: 10 Jan 2003

Posts: 1146

PostPosted: Sun Feb 29, 2004 6:27 am    Post subject: Reply with quote

Jeremy posted a few days ago that the servers have been getting hammered by 'bots'.
_________________
"Hi, I'm Moe, or as the women know me - Hey! You in the bushes."
-Moe, The Simpsons
Back to top
View user's profile Send private message Visit poster's website
rickrich
Geocacher


Joined: 06 Jul 2003

Posts: 673

PostPosted: Sun Feb 29, 2004 11:05 am    Post subject: Reply with quote

First, given that a person is seeking a certain dataset from the site, a bot will generate less total load than the same queries posed by the 'clicky-clicky-scroll-clicky-clicky' process. In the case of non-PQ queries, this is because the bots do not request any of the images on the site. In the case of PQ-queries, this is because with a bot there is no need to schedule daily PQ's at all, they can be easily performed only when you need them.

Second, bots are used because Jeremy has refused to provide features that people have repeatedly asked for. Among the things I can think of off the top of my head are:

2a) PQ's are limited to 500 caches, so you either have to manually generate multiple overlapping PQ's to get the entire metro area (increasing server load), or simulate GPX queries by scraping thru the web interface. This is exacerabated because Jeremy has provided only circular regions, not rectangular regions. Rectangular regions can be non-overlapping and do not require great-circle calculations. To accomplish the same with circular regions requires a 44% overlap and great circle calcs.

2b) PQ's are limited to 5 logs, which is often not enough. Once again, you are forced to use (either manually or with a bot) the web interface to see a sufficient number of logs.

2c) New cache notifications. If you want to be an FTF, you need timely new cache notifications. An email on Thursday does not cut the mustard. But AFAIK, gc.com does not offer instant email notofication when a new cache is placed. So people are forced to poll the site with a bot multiple times per day.

2d) Stats. Need I say more?

2e) Off-site cache listing and log backup.

2f) Search for all "latest logs" in an area so on a rainy day you can read up what your fellow cachers have been doing.

Third, Jeremy has made a number of changes in the last 9 months that make a bots job harder to do. The timing of these changes makes it pretty clear that he is aiming these changes at bots and not to provide new functionailty. Bots aren't going away until gc.com provides what people need from the site. So he has reaped what he has sown.

Fourth, Jeremy has not paid attention to the clicky-clicky usage pattern of humans accessing the site, and has not optimized the site for the access patterns that humans will generate. This generates additional and unneeded load, which is not caused by bots at all, that he could correct with a redesign of the site. Especially when in "logging caches mode", a large number of unnecessary database queries and page loads are generated.

Fifth, there was a fifth point, but I got brain block and can't remember it right now. I reserve the right to edit this post when my brain recovers.

Fifth (remembered now), the latest feature of modern web browsers is to pre-fetch pages in advance of the user requesting them. Mozilla does this, and I'm told that IE6 has something similar but not as good (I could be wrong about IE6). Its entirely possible that some of the increased load on the site is due to the growing popularity of this browser feature. Unfortunately, this browser feature looks and acts a lot like a bot, but without the precision targetting of just the data required that only a well-written bot can do.

Sixth, Jeremy is as guilty of leaching as the people he condemns. For example, when a new cache is placed, the map image comes from the US Census Tiger map server. New caches generate lots of website activity, and he is generating load on the Census bureaus server for his commercial purposes. The TMS people have requested that commercial sites do not do this. Hypocrit? You decide.

Jeremy wants to run the site "for profit". Unfortunately, this has forced him into a site design that does not scale very well other than by throwing hardware at the problem. He has made his bed and now he must lie in it. If the profits of the site are not sufficient to support the required new hardware investment every 3 months, then perhaps he should rethink his business. An open database would allow for a very scalable design, potentially harnessing the power of the computers of every geocacher.

So am I part of the problem? You decide.... I run a 'newest caches' bot every day at 5AM that scrapes 2 web pages. I run a 'stats' bot twice a week (Monday and Friday mornings). I run an FTF bot every hour that scrapes 1 web page. I don't use daily scheduled PQ's, rather, I run a bot-generated on-demand PQ 10 minutes before I leave the house. On average I run this about twice a week (some weeks never at alll, some weeks more than twice, depends on whether I am going out or not). I am a paying member, and I don't hunt MOCs or TBs. Every few months I use a bot to do an incremental backup of caches I have placed or found, including logs. This is always run during the week in the wee hours.

I'll probably end up getting banned from gc.com for posting this, but it had to be said.

-Rick
Back to top
View user's profile Send private message Send e-mail
s4xton
Past MnGCA Board


Joined: 23 Mar 2003

Posts: 1070

PostPosted: Sun Feb 29, 2004 8:18 pm    Post subject: Reply with quote

RickRich - I'm with you 100%.
Back to top
View user's profile Send private message Visit poster's website AIM Address
15Tango
MnGCA Board


Joined: 17 Dec 2002

Posts: 857

PostPosted: Sun Feb 29, 2004 11:21 pm    Post subject: Reply with quote

In his gc.com forum posts on the same topic, it seems to me that Jeremy claims it's a bug in the system, and not related to the amount of users on the system. But, when I cache during the week, I'll go caching, then go to work at 3:30pm, and write my logs after I get home at 2:30am, and have no problems. Then when I cache on Saturday or Sunday, I'll write my logs that evening when everyone else does, and keep getting the Server to Busy thing. Last night, it took me over two hours to log 11 finds and 2 DNFs--good thing Saturday is a bad TV night and I didn't have any other plans. I'm also a Charter Member, why do I get chosen as the deadlock victim?
_________________
There comes a time in every young boy's life when he gets an irresistible urge to seek buried treasure.--Mark Twain
Back to top
View user's profile Send private message
rickrich
Geocacher


Joined: 06 Jul 2003

Posts: 673

PostPosted: Mon Mar 01, 2004 12:52 am    Post subject: Reply with quote

Yes, I see that Jeremy has now backed off from blaming the 'bots' for his troubles.

He even started a new thread with essentially the same subject line as the old thread. No doubt he's hoping the old thread will fall off the first page and he can save some face.

Now, the data is that an "astounding" 50,000 logs were created during the last week, and this may be the source of his troubles. Even if everybody decided to log those 50,000 logs/week in just 4 hours on Saturday evening and 4 hours on Sunday evening that would amount to a transaction rate of less than 2 logs per second for the servers to handle. This really is not a large load.

I am glad to see Jeremy focusing now on the real source of the problem, which is the architecture of his system and the programming behind it. Unfortunately, he seems to be applying bandaid fixes (bailing water from a boat with a teaspoon). I expect he will get things under control for the time being, only to have this problem reoccur later this year. Its an architectural issue that he's going to have to face eventually. I saw he was looking for resumes Smile

I also saw that robertlipe gently thru down the gauntlet for Jeremy to open up the source code so that more eyeballs could work on the problem. Nice chess move Robert! They will find Bin Laden before Jeremy opens his code, though.

If we go a period of time with no trouble and then the trouble surfaces again, I bet he will blame the 'bots' first. Anybody wanna take the bet?

-Rick
Back to top
View user's profile Send private message Send e-mail
robertlipe
Geocacher


Joined: 01 Mar 2004

Posts: 15

PostPosted: Tue Mar 02, 2004 12:51 am    Post subject: Reply with quote

rickrich wrote:
Even if everybody decided to log those 50,000 logs/week in just 4 hours on Saturday evening and 4 hours on Sunday evening that would amount to a transaction rate of less than 2 logs per second for the servers to handle.
I did the math on that, too. (Even if you throw in some skew for the earth's rotation, the habits of people preferring to sleep when it's dark, and the fact that over 23% of the worlds caches exist in one time zone, and it's still not a lot.) If the site was well designed, it should be able to run on a nice laptop. If you put it on a Real Computer, there's just no reason it should be huffing and puffing.
Quote:
I expect he will get things under control for the time being, only to have this problem reoccur later this year.
History has proven this to be true.
Quote:

If we go a period of time with no trouble and then the trouble surfaces again, I bet he will blame the 'bots' first. Anybody wanna take the bet?
History has proven that's a sucker bet. Smile But it's amazing how much backlash there is in the forums every time it comes up against those evil page scrapers.

Oh, and history has proven that posts like mine and Rick's will be utterly ignored.
Back to top
View user's profile Send private message
rickrich
Geocacher


Joined: 06 Jul 2003

Posts: 673

PostPosted: Tue Mar 02, 2004 2:47 am    Post subject: Reply with quote

Hi Robert! Nice to see you step into our forums. Team MnGCA, meet robertlipe, an alpha-geek from the great state of Tennessee. He's a virtual cache you should seek on your trip to MO this spring Smile.

Robert is the team leader for gpsbabel, a fine piece of open source software thats at the heart of many page scrapers and many of the programs that feed your PDAs and GPSrs their waypoints.

Also, it looks like we will have a new leaderboard leader after Friday morning's scraping Smile.

-Rick
Back to top
View user's profile Send private message Send e-mail
Marsha and Silent Bob
Past MnGCA President


Joined: 02 Sep 2003

Posts: 6261

PostPosted: Tue Mar 02, 2004 6:54 am    Post subject: Reply with quote

robertlipe wrote:
Oh, and history has proven that posts like mine and Rick's will be utterly ignored.

They actually defend him to the end of the earth. They keep ignoring the fact that he has already blamed the bots for causing this issue two weekends ago. HE LIED TO YOU AND YOU STILL SUPPORT HIM. They keep having excuses like "you make your own site", "he's now asking for resumes", "he never expected it to become this big", etc.

I have been pretty disappointed in the past but this was ridiculous. It's just the fact that his supporters somehow outweigh those that are against him (at least vocally... who knows how much work their forum moderators do against the cause though).

Silent Bob
_________________
Sad state of affairs.
Back to top
View user's profile Send private message
robertlipe
Geocacher


Joined: 01 Mar 2004

Posts: 15

PostPosted: Tue Mar 02, 2004 8:42 am    Post subject: Reply with quote

Thanx for the welcome. If me joining the forum messes up your local stats site, I'll gladly unsubscribe or ask to be excluded or whatever. In TN we already have a lovely stats site so I don't need to skew your bell curves. (As an aside, you guys should get a piece of that action.) Rick and I have been friends and were coworkers for many years. I probably helped bait him into geocaching.

You're exactly right, Silent Bob. The domination of 'fan boys' in that crowd often gives it a ring of the Ministry of Truth. There are lots of things spoken and repeated as gospel that real techies know hold no water at all.
Back to top
View user's profile Send private message
Moe the Sleaze
Geocacher


Joined: 10 Jan 2003

Posts: 1146

PostPosted: Tue Mar 02, 2004 8:49 am    Post subject: Reply with quote

Anyone care for a little cheese with their...

Laughing
_________________
"Hi, I'm Moe, or as the women know me - Hey! You in the bushes."
-Moe, The Simpsons
Back to top
View user's profile Send private message Visit poster's website
s4xton
Past MnGCA Board


Joined: 23 Mar 2003

Posts: 1070

PostPosted: Tue Mar 02, 2004 8:56 am    Post subject: Reply with quote

robertlipe wrote:
Thanx for the welcome. If me joining the forum messes up your local stats site, I'll gladly unsubscribe or ask to be excluded or whatever.


Thanks for joining. You don't mess up anything. RickRich's script simply takes the userlist from these forums and finds their geocaching profile... so mtn-man, etc. are in our stats as well, while some Minnesota cachers that have never been to these forums are currently excluded... no big deal.

It's good to have more people around that are knowledgable on such topics.

-Aaron
Back to top
View user's profile Send private message Visit poster's website AIM Address
Marsha and Silent Bob
Past MnGCA President


Joined: 02 Sep 2003

Posts: 6261

PostPosted: Tue Mar 02, 2004 9:03 am    Post subject: Reply with quote

Moe the Sleaze wrote:
Anyone care for a little cheese with their...

I wouldn't exactly call it whining. I call it stating facts. I am less concerned about the issues the site has been having. I am quite concerned w/the fact that Jeremy blatantly lied about the cause of the problems and those that support him continue to do so while making poor excuses and blatant denial of the fact that he lied about the problems last weekend.

Silent Bob
_________________
Sad state of affairs.
Back to top
View user's profile Send private message
s4xton
Past MnGCA Board


Joined: 23 Mar 2003

Posts: 1070

PostPosted: Tue Mar 02, 2004 9:06 am    Post subject: Reply with quote

Moe the Sleaze wrote:
Anyone care for a little cheese with their...


Well, I might be a guilty recipient for some cheese... but there are a lot of us who are taking action and doing a lot to improve and bring positive change to the technical side to geocaching. Because of Groundspeak and their policies, a lot of efforts have cropped up such as RickRich's scripts, Buxley's Waypoint, Navicache, Scout's stuff, ubercaching.com, Keyori, opencaching.org and other related efforts such as gpsvisualizer and gpsbabel to making geocaching more usable, accessable and in the end more fun for geocachers. Part of this is to create a geocaching world that's not dictacted by Jeremy Irish and his pocketbook. I wouldn't have anything against him if he listened to his users (which would in the end bring less burden on him and his servers), instead of listening on how he can capitalize on our free sport. I don't mind people making money - but I don't like it when it's at the expense of usability, and when they're liars as he demonstrated again recently.

Point again, we're not just whining.

-Aaron
Back to top
View user's profile Send private message Visit poster's website AIM Address
Display posts from previous:   
Post new topic   Reply to topic    MnGCA Forum Index -> Geocaching Websites All times are GMT - 6 Hours
Goto page 1, 2, 3, 4, 5, 6  Next
Page 1 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group
Geocaching Cache Icons, Copyright 2009, Groundspeak Inc. All rights reserved. Used with Permission.