Thanks for your help above guys, it set me on the path to finding what I think is the most simple solution. After brushing up on my win32 api I read that a modal window is basically a window which has disabled its parent, so I found the win32 api function IsWindowEnabled and used that on the main application window. I found that the main application window is disabled when a response window is open in the application, whether the response window has been opened by the main window or from one of its children windows. My app_manager class has a function which returns the main application window, so my whole function to find out whether a response window is open is pretty short.
// Declare the Local External Function
Function boolean IsWindowEnabled(ulong hWnd) library "user32.dll" alias for "IsWindowEnabled"
// My function of_isEnabled()
// returns false if a response window is open
ulong lul_handle
boolean lb_enabled = false
lul_handle = Handle(gnv_app.of_GetMain())
if lul_handle > 0 then
lb_enabled = IsWindowEnabled(lul_handle)
end if
return lb_enabled