ThriftHunting

[figured if I was going to right a book (oops) I should move this to a better topic area so it doesn't clutter the New People area...]

On the HTTP header, (BTW I do want to try out the mobile version page with my new phone/PDA...)

I meant what your "backend" query sends to Amazon/Half, etc. to get the price- not about your Mobile site version responds to me with.

I am only semi-literate when it comes to website coding, so I will try and outline what I think in case anybody else reads this later it will hopefully makes some sense. Real code monkeys will see the errors in what I say though I'm sure. Here goes.

Some sites (like here!) have a mobile version page you can get when you request the right path (e.g. www.site.com\mobile or ...\pda).

Some other major websites (Google, Amazon, etc.) automatically detect when you are on a phone or PDA and they return their mobile version as appropriate (you don't ask for the special ...\mobile URL but you get it anyway).

As I understand it, they do this because each browser inserts several invisible bits (headers) into the HTTP request it sends to the server- they announce what browser version they are- and the website application is programmed to respond differently for mobile versions. Many of us have heard of the HTTP GET (ask for data) and HTTP POST (put data) but there are plenty of other headers too. I saw this... http://en.wikipedia.org/wiki/List_of_HTTP_headers and I think the "User-Agent" tag is the one I am thinking of.

When we use a regular browser like Firefox you can't change what headers it sends- it's all automatic and invisible. But if somebody is writing a program or routine to go pull info from another website... well, I think you can code your bot or whatever to send any headers you want to program in.

Then I looked and found this... http://www.ericgiguere.com/articles/masquerading-your-browser.html which describes what I mean pretty close. It shows how to monkey with these even for standard browsers too... looks like there is all kind of stuff when I googled for "mobile browser detection".

So if your program sends the request to Amazon (and the others?) and you embed the User-Agent header with a mobile version ID (like Windows CE?) then Amazon automatically replies to you with a simplified "mobile" version result- instead of the cluttered normal page. The mobile version of Amazon does not have all the images, ads, etc. the normal page has. I think as a result of that, the answer you want in our case here (just the $$) comes back to you faster. Not that the site seems slow... it's certainly faster than I can type in a UPC code (or probably scan it). But it occurred to me to wonder if that kind of think could possible speed up a List Batch search.

So... I'm curious, what you think? I know there is always a major danger to "clever ideas" when I don't know enough about something- it only seems clever because I don't know enough to see why it is idiotic. I know this may not apply at all to the way you actually execute those backend queries, but I got kind of curious when I saw this great thing.

[[ I was thinking of trying to work something up (or have one of my code monkey friends help me figure out how to do it) that I could run on my desktop that would pull ALL the Amazon Used prices for ONE item (not just the lowest price) and report several things like Average, how much the 5th and 10th listings were (ranked by cost like cheapest to most expensive), by Conditions, etc.. What I find is the lowest Used price might be for "Used-Acceptable" condition with say a cracked case and no manual- and be $5... when the 2nd listing is for "Used-Good" and is $20. More common is some CD's are listed for .01 (cheapest) but there is only 2-3 at that price and then it jumps to $6 (fourth listing) and higher. That affects whether I should buy an item for $1!... or if I did whether I should donate it or list it. It just takes 2-3 minutes per item to really check the listing prices in depth, and evaluate it but... multiplied by a box of 50 CD's... So it occurred to me to wonder if this technique might be useful to me. But if it's not a good idea, I want to understand a little "why not"... so maybe then I could try something better. Trying to go mobile with this seems to daunting though. Part of why I think this site is awesome!]]

Hope that makes more sense. Probably just put everybody to sleep though. *<80)

Hi again,

So I reread your reply and can see your point. And as always a simple goal helps make for a simple (and supportable) design. That's a principle I try and use at work as well...

On a related quest... kind of follows up on this idea. I am looking at maybe trying Excel web queries if I want to have a "tracking board" for a short list of video games. Maybe track the top 20 sellers in each category, etc. kind of thing which is a little different than what the tool here is supposed to do. If I get crazy I might try and pull the same kind of data from Amazon, Ebay, Gamestop, etc. for arbitrage between platforms? Just a crazy idea for my spare time...

Here is the best example of what I posted earlier. Amazon has this URL which only returns the simplified "mobile" page results. You can see the results similar to my original idea for "http header browser version" instantly.

http://www.amazon.com/gp/aw/

I am certain Excel will be faster grabbing the details through this URL/method- it's not necessarily how fast Excel could "ignore" the influx of data on the regular page... but as this method provides so much less data from the site itself it comes through much faster. I imagine the database lookups are so much simpler on their end. Pop in the UPC and presto the data comes back immediately, and using that URL approach is easier than trying to post headers (not sure Excel can do that).

So that should work good for my purpose for my little experiment... just food for thought.


Cheers,
John