Question about non-compete clauses in book contracts

There’s little in a writer’s contract that’s more problematic in the long run than the darn “non-compete clause”. It’s a classic problem if you think about it because from the publisher’s perspective, they want to lock you down as broadly as possible, but from the perspective of a writer, you really want to have it as microscopically narrow as possible so that you retain the most freedom you can. The result is, unsurprisingly, tension and, often, a monster lurking in the closet.
How do you know that Publisher Y is going to view your new book with Publisher X as falling into the non-compete zone? Have they contacted you, or did you contact them proactively?
The thing of it is that in many ways non-compete clauses are unenforceable because they prevent you from capitalizing on your domain expertise. If you are an expert on search applications and how to use them to unearth valuable data, then they are blocking you making a living. Not good. However, that doesn’t mean that a court would view it the same, so I think it’s smart for you to be cautious!
What I would recommend is notifying X that you have this potential conflict ASAP, and then calling up the acquisitions editor at Publisher Y and requesting that they send you a formal release, permission for you to publish a book on search tools with Publisher X. Worst case, they might ask for their 30-day right of first refusal option, in which case you send them the proposal for the book, they waste a few weeks, they reject it, THEN you can legally go back to X.
Update: The author sent in this further clarification of the situation…

Truth is, Dave, I began writing the new book without giving much consideration to the no compete clause. My editor at Publisher X brought up the issue. I contacted Publisher Y and gave a list of the differences between the two books. But, the legal department decided that the two books were competing. The titles of some sub-topics in both books are the same.
I inquired with a lawyer friend who is not a specialist in book contracts, but who does understand contracts in general. His
comments were –
1. The law applied to the contract depends on the address of the publisher. So, California law would apply to a contract with Publisher X.
2. The burden of clarity of the contract lies with the publishers, since they draft the contract and authors can negotiate only parts of the contract.
3. The no compete clause is reasonable since it is in the publisher’s interest to prevent other books which are similar and will cut into the sales of the original book.
I think this makes sense from the author’s point of view as well. Writing two books with substantially similar content is likely to be noticed by readers and not appreciated.
4. The no compete clause does not prevent an author from writing a book that is “loosely related” to an earlier book. Such a book would not be directly competitive with the old book.
5. A publisher may have a case even though the content of a new book is substantially different, but yet competes for the same audience as the old book. So, a publisher could state (and may be prove) that the new book’s publication is cutting into the sales of the
old book.
This is my opinion. I think there is a difference between a second edition of a book that has been re-packaged as a new book and a truly new book with substantially new matter that does not fit in a second edition. Both books may be related to the same topic, but in most computer-related topics, there is enough new material to create a new book, if the two books are published two or more years apart.
I think it is extremely likely that any publisher will give permission to publish another book that even seems to be somewhat similar to an earlier book. It is in their best interest to deny permission, rather than take the risk of granting permission and paying the penalty later.
So, it would probably be much easier to negotiate a new and related book with the same publisher than another publisher.

I agree with your assessment, that publishers are better off to err on the side of saying “no” rather than risk the chance that another title by the same author on substantially the same topic will compete effectively in the marketplace and cause a drop in sales.
I’m afraid that in this situation, especially if you have similar chapters or subchapters, that your best bet is duck out of the relationship with Publisher X and go back to Publisher Y to see if they’d like to publish the book you’re working on. It’s too bad – the gang at Publisher X [note to readers: the author shared actual publisher names with me, I just changed them to protect his privacy] are a terrific group and quite pleasant to work with!
I hope that helps you understand how to navigate these dangerous waters!