WebSpider

Today's assignment is WebSpider. My spider is here.

---

All three tasks accomplished.

HttpUnit

I used HttpUnit to create fake servers for testing my spider, so I learned about the server parts of it as well as the client parts. It's a actually nicer on the server end, and I wouldn't mind using it for that again. (But it's terrible on the client side.)

Logging

I didn't spend very much time on logging. I just grabbed something from Logger.getLogger and used setUseParentHandlers to turn it on and off. That seems a bit crude, but it works.

Kevin English posted some nice code to suppress all those HttpUnit stack traces. It's not really a part of logging, but it cleared away the screen so I could see my log.

Java

I had a very interesting time with Java on this project. HttpUnit (on the client side) has a bad habit of declaring checked exceptions that it never throws, then throwing many different unchecked exceptions that crash the program.

At the same time, I was trying to wrap part of the program in an Iterator for the silly reason that it made logical sense. Java won't let you add checked exceptions when you're implementing an interface, so I couldn't just propagate the checked exceptions knowing they didn't actually happen.

I ended up with some dead code to satisfy the compiler, a lot of try/catch blocks covering different levels of nesting (if it fails in the outer loop, continue the outer loop; if it fails in the inner loop . . .) and pre-fetching to make sure that next always worked when hasNext was true.

Basically, several rules of Java that are supposed to stop things from going wrong actually helped them go wrong and then got in the way of fixing them.