Something Evil About Robots.txt I Didn’t Know

Quick background: A robots.txt file on your website will tell search engines and other bots that obey the robot exclusion standard what files and folders they can and can’t index, or whether they can access the website at all.

I’ve been working on the robots.txt file at work the last few days.* Once the file had the bots I wanted to exclude I decided to run it through a robots.txt validator.

Boy did I learn a few things. It turns out that you should put robot exclusions at the top and directory and file exclusions below. There were also a few minor formatting issues that I’m not sure really mattered.

There was one, however, that was a shock. Let’s say you’ve got a folder called “video”. There’s a huge difference between these two disallow statements:

Disallow: /video/
Disallow: /video

The first example with a trailing slash tells robots not to index anything in the video directory. So far so good. The second example without a trailing slash tells robots not to index anything in the video directory, or any file at the root level with video at the beginning of the filename.

Without the trailing slash, you would exclude /video.html, videoplayer.aspx – you name it. Anything at the same level of the directory structure that begins with video. You can get into trouble in a hurry if you leave the backslash off of the disallow directive.

* What prompted the work was all of the bots that kept showing up in our error files. One of the worst? The Internet Archive Bot that collects pages for the Internet Archive. It would generate hundreds of errors a day. When I looked around at bot ban lists the IA bot showed up over and over. You’d think Internet Archive would  have worked the bugs out of their bot by now.

Posted in Ecommerce, Tech | 2 Comments

DSLR Video – “Higher Calibre, Higher Mindedness: The Story of YoGun”

So last weekend Melissa and I went to the Knoxville 24 Hour Film Festival. The idea was for teams to shoot a four minute in 24 hours. The required elements were:

  • A bounty hunter.
  • A large body of water.
  • The phrase “So there I was minding my own business.”
  • Fire.

This video ws about YoGun, the fictional merger of yoga and guns and the inner peace one acquires from shooting. Guns and video, what’s not to like?

“Spring Rain, congratulations. You have progressed from level one to level four, revolvers and the associated calibers. You have now graduated to Zen level five, semi-automatics.”

Posted in Misc | Tagged , | Comments Off

This is Your Train on Drugs, UK and Detroit Editions

High speed rail scheme cost to double to £80bn, economists warn:

HS1, the high-speed rail line that connects the Channel Tunnel with London, was initially expected to cost £1billion. The final bill was around £11billion.

The London Underground’s Jubilee Line extension, the biggest rail project before HS1, came in at four times the original estimate in real terms.

And unlike buses, trains require the destruction of everything along their route:

Even though the first train is not due to run along the new line until 2026, values of homes close to the route have already fallen by as much as 40 per cent. Estate agents have said that properties up to a mile from the route are being blighted by the proposed line, with some close to the proposed line failing to sell at any price.

Apparently the UK wants to be Springfield to Detroit’s North Haverbrook.

Everyone’s heard about Detroit’s financial problems. One of the many failed attempts to revitalize their downtown (and to funnel taxpayer money to political cronies) was a train system, the Detroit People Mover:

The Mover costs $12 million annually in city and state subsidies to run.[9] The cost-effectiveness of the Mover has drawn criticism.[10] In every year between 1997 and 2006, the cost per passenger mile exceeded $3, and was $4.26 in 2009,[11] compared with Detroit bus routes that operate at $0.82[11] (the New York City Subway operates at $0.30 per passenger mile). The Mackinac Center for Public Policy also charges that the system does not benefit locals, pointing out that fewer than 30% of the riders are Detroit residents and that Saturday ridership (likely out-of-towners) dwarfs that of weekday usage.[12] The system was designed to move up to 15 million riders a year. In 2008 it served approximately 2 million riders. In fiscal year 1999-2000 the city was spending $3 for every $0.50 rider fare, according to The Detroit News. In 2006, the Mover filled less than 10 percent of its seats.[12]

Among the busiest periods was the five days around the 2006 Super Bowl XL, when 215,910 patrons used the service.[13] In 2008, the system moved about 7,500 people per day, about 2.5 percent of its daily peak capacity of 288,000.[14][15]

Under-utilized and overbudget is a pretty good summary of recent urban trains.

PreviouslyThis is Your Train on Drugs: CA Train Versus Endangered Species, Asthmatic Children

Posted in Economics, European Union | Tagged | Comments Off

Good and Evil, and Commerce and Capitalism as a Way to Cure Poverty

“In fact, Bono, C. S. Lewis has a great quote which I love: ‘When a man is getting better, he understands more and more clearly the evil that’s left in him. When a man is getting worse, he understands his own badness less and less.”
Jim Daly, interviewing Bono

That was in the context of interviewing Bono, who recently said “Aid is just a stopgap. Commerce [and] entrepreneurial capitalism take more people out of poverty than aid. We need Africa to become an economic powerhouse.”

A crazy new idea for Africa and most third world nations is to observe the rule of law, recognize constitutional rights, engage in democracy, bring women and minorities equal rights, and engage in free market capitalism. Historically, that’s a hell of a winning formula. It worked for most first world countries.

India and China have improved their lot just by practicing that last part, free market capitalism. Both of them – but especially China – have a long way to go on the other parts that have to do with human rights, but the little bit they’ve done so far has improved the lifes of several billion people. Foreign aid and economic redistribution could never have done what capitalism has done for those people.

PreviouslyBreaking News: Bono and Matt Yglesias Mugged by Economic Reality

Posted in Economics, Political Survival Kit, Quotes | 1 Comment

Word of the Day – Predicate

My fourth grader asked me for help with her English homework, which was all about predicates. I had completely forgotten the difference between objects and predicates and had to look it up. The lesson? Once you graduate high school you’ll probably never use this stuff in daily life and will eventually forget it, but it is nice to know.

We watched a School House Rock video on predicates, but this page had the best explanation.

One of the two main parts of a sentence or clause, modifying the subject and including the verb, objects, or phrases governed by the verb. Adjective: predicative.

In both grammar and logic, the predicate serves to make an assertion or denial about the subject of the sentence, as in “Merdine sneezes” and “Gus never smiles.”

Previous WOTDBliss Point

Posted in Word of the Day | 1 Comment

Still Alive and I Have a Job No Less

No posts lately because I’ve been busy at a new full time gig, which I like a bunch. I needed a job and they needed me badly. ‘Nuff said.

Meanwhile, I’m putting out the videos I’ve been working on for a while. Here’s one.

Hootchie Cootchie Man

Posted in A&E, Home Life | Tagged , , | 3 Comments

“Kids and dogs: If you’re having a baby, do not get a puppy.”

The One Thing No One Tells You Before You Have Kids – Don’t get a dog. People in comments are raking her over the coals for dissin’ the dog.

I don’t think her problems were caused by the dog. Read between the lines and you discover that at the time she got fed up with the dog she had three kids under the age of five. That’s a lot of butts to wipe.

We had two kids under the age of three, plus a dog and three cats. We were in our mid-thirties and couldn’t put off kids too much longer, my wife was in school which actually worked out great with kids, and so we decided to go ahead and have a second. It wasn’t easy, though I can’t imagine having three that young. A year or two later my mom couldn’t live alone any more due to failing health and being mostly blind, so she moved in with us, and after she developed Alzheimer’s things got much more difficult for a few years. Thus endeth my tale of woe.

So I don’t think the lesson is don’t get a puppy. The lesson is that if you can it’s nice to put a little space between the kids’ birthdays.

Posted in Home Life | 1 Comment

Detroit Files for Bankruptcy

Trying to pay for stuff you can’t afford will do that to you:

It’s the largest municipal bankruptcy in U.S. history, dwarfing Jefferson County, Ala.’s $3.1 billion sewage district restructuring.

In June 2012, the City of Stockton became the largest-ever city to file for bankruptcy, at the time.

The Motor City faces $20 billion of long-term liabilities. The Wall Street Journal’s Matt Dillon says those holding onto $11 billion in unsecured debt are basically staring into the abyss, facing the prospect of getting next to nothing from the city’s obligations.

The pension funds want to block Orr’s attempt to drastically reduce the amount of benefits owed to current and former city workers.

Pension funds for unionized city employees are one of the reasons Stockton went bankrupt.

Posted in Economics | Tagged | Comments Off

Consensus Changes on Salt

CDC says Americans still consume too much, but studies show no benefit in reducing salt

Richard Feynman said that one of the advantages of the scientific method has over religion is that science has the freedom to be wrong. Religious institutions are reluctant to ever admit error. The scientific method lets experiments and new data supercede previous ideas.

That works well when science is an ivory tower institution. The problem comes when bad science is applied to real world problems or used to set public policy. By the time science corrects itself the damage has been done, or how long it will take the new science to replace the old.

Posted in Science | 3 Comments

I had wondered what was going on in the peak oil world

Oil and gas production in North America is booming and we’re past the usual estimates for peak oil. Now it turns out that one of the peak oil advocacy sites is shutting down.

Posted in Environment | Tagged | Comments Off

DNA May Solve Boston Strangler Case Once and For All

Mysteries without closure bug me, so I’m glad to see this: Mass. DA: DNA links DeSalvo to Strangler victim.

Posted in True Crime | Comments Off

Update on the Crispy Fried DeLorean

Joey-Paul Gowdy writes:

Howdy, a friend sent me the link to your article:

I purchased that car in Sept. of 2009

Did a lot of work to it… current progress:

It’s a mighty good restoration of a burned car.

Posted in Nifty | Comments Off

Double Edged Safety Razors – The Blades

Double Edged Safety Razor Cartridges

For various reasons, I’ve decided to try shaving with an old-fashioned, double-edged safety razor. One part of the appeal is price. Most plastic cartridges razors cost around $2/cartridge, or $1 at the Dollar Shave Club.

At local grocery stores, plain old razor blades were 10 for $15, which is nuts. I bought a 120 pack of blades on Amazon for $19, so about 16 cents per blade. Can’t beat that.

Breda asked if I also bought Band-Aids in bulk. I didn’t (smartypants), but after shaving with one of these blades I walked straight to the kitchen and wrote styptic pencils on the grocery list.

But what a close shave. Here’s hoping that over time I’ll either get much better at shaving with a DE, or slowly, bloodily carve my face into a shape that’s better for DE shaving.

Posted in Home Life | Tagged | 4 Comments

New Industry Standard for IS/VR Lens Performance

Canon offers IS and Nikon offers VR – technologies that reduce picture blur when handholding a camera. There’s always been some debate about how well those systems worked, and the companies keep coming out with new versions that area supposed to be more effective, but there’s never been a standard for measuring effectiveness until now.

Nikon has published the results of their CIPA testing. Results range from a low of 2.5 stops to a high of 4.5 stops. 4.5 stops means that the image-stabilizing technology is as effective as increasing shutter speed by 4.5 stops. So a photo taken with VR at 1/60th of a second would be equivalent in sharpness to a photo taken without VR at 1/1500th of a second.

VR (and Canon IS) mean you can get a sharp picture using shutter speeds that are slow enough to get a good exposure in a broad range of lighting conditions. They also mean you can get the shutter speed down in the range where you can use a flash at full FP sync – typically 1/200 to 1/250th of a second.

P.S. VR/IS lenses are also great for handheld video. You shoot video at a shutterspeed of 1 over twice the frame rate – so 1/60th of a second for 30 fps video -  so you can’t get sharp images by using a quick shutter speed.

Posted in Photography | Tagged , | Comments Off

Former East German Stasi Head on NSA Surveillance

“It is the height of naivete to think that once collected this information won’t be used. This is the nature of secret government organizations. The only way to protect the people’s privacy is not to allow the government to collect their information in the first place.”
Wolfgang Schmidt

Posted in Political Survival Kit, Quotes | 2 Comments