I got asked by two separate friends of mine just within a few days of each other if I had any recommendations for what DOCTYPE tag should be used in SharePoint. Honestly up until that point, I had not put a lot of thought into outside of the attitude of look up whatever A List Apart recommends and use that. So I delved into it a bit more and now I do have an opinion on the matter.
A little info first, DOCTYPE is short for Document Type Declaration. A DOCTYPE associates a web page with a Document Type Definition (DTD). The DTD is the definition of the particular version of HTML or XHTML you are using in your site and is what will be used for rendering of the web page. For DTDs, there are generally 3 types, strict, transitional and frames.
Strict means there isn't presentational elements in your code, such as formatting and alignment. Instead everything has be to controlled separately in the CSS. Transitional allows both CSS and presentational elements in the code. Frames, well is for frames and if you are still using those we need to chat. :-)
Based on your DTD and what browser you are viewing your site in, your page will render in Standards mode or Quirks mode. Standards mode renders the page based on the DTD you defined. Quirks mode will render the page the best it can while emulating older browsers. If you don't define a DTD to use, the browser will default to Quirks mode.
So what about SharePoint? First off, note that default.master does not specify a DTD! The publishing master pages (blueband, blackband, etc) use the HTML 4.0 Transitional DTD that renders in Standards mode.
For my base master pages, I put in a DOCTYPE that uses the HTML 4.0 transitional DTD that will render the site in quirks mode. Why? It is the lowest common denominator and appropriate for the general use of the master page. This is not to say this is the DTD you should be using for your site. Ultimately, you need look at the site's target audience and purpose and select a DTD based on your individual site needs.
However, I do think Transitional DTDs should be used since there is a lot of rendered code with out of the box SharePoint that you don't have control over, therefore your page will have presentation elements in the code and not solely use CSS for formatting. That and Transitional is best when you use tables for page layout. Tables vs. CSS positioning wars aside, out of the box SharePoint makes heavy use of tables for layout so stick with the transitional DTDs.
I personally don't use the XHTML DTDs because I have yet to see justification for using them, and I get a bit worried about the fact that XHTML 1.0 won't be supported in XHTML 2.0 and vice versa. It seems to me if you go with a XHTML DTD, you are limiting yourself. But again, this goes back to what does your site need, not what makes me nervous.
Another important thing about DTDs, is you need to select one BEFORE you delve too deep in developing your user interface. The DTD you specify in the DOCTYPE tag will affect how your page is rendered, so don't write a bunch of HTML and CSS and then apply or change the DTD, many of your page elements could go out of whack and you will have to rewrite some of your code.
Here are the two DTDs I think are best for SharePoint:
HTML 4.0 Transitional in Standards Mode (use this one if you can):
<!DOCTYPE html PUBLIC "-//W3C//DTD html 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
HTML 4.0 Transitional in Quirks Mode:
<!DOCTYPE html PUBLIC "-//W3C//DTD html 4.01 Transitional//EN">
Here are some additional resources:
UPDATE 1/25/08: Well DOCTYPES must be in the air, Waldek Mastykarz just posted about DOCTYPEs as well, and recommended the use of XHTML 1.0 Strict or Transitional DTD. He talks about his reasonings and the benefits, drawbacks and challenges with this approach. For a different viewpoint, read his article here.
UPDATE 3/10/08: Randy Drisgill has tested SharePoint in the next version of Internet Explorer, IE8, and has a recommendation to include the meta tag that forces IE8 to render the web page based on IE7 parameters.
<meta http-equiv="X-UA-Compatible" content="IE=7" />