Applications of elliptic operator theory to the isotropic interior transmission eigenvalue problem

Let us recall that $\lambda \in \mathbb {C}$ is called an interior transmission eigenvalue (ITE) if the homogeneous problem,

$$\begin{equation} {-}\Delta u - \lambda u =0, \quad x \in \mathcal {O}, \quad u\in H^2(\mathcal {O}), \end{equation} \tag{ 1 }$$

$$\begin{equation} {-}\Delta v - \lambda n(x)v =0, \quad x \in \mathcal {O}, \quad v\in H^2(\mathcal {O}), \end{equation} \tag{ 2 }$$

$$\begin{eqnarray} &&u-v=0, \quad x \in \partial \mathcal {O}, \\ &&\frac{\partial u}{\partial \nu } - \frac{\partial v}{\partial \nu }=0, \quad x \in \partial \mathcal {O}, \end{eqnarray} \tag{ 3 }$$

has a non-trivial solution. Here, $\mathcal {O}\subset \mathbb {R}^d$ is a bounded domain with a C^∞-boundary, $H^{2}(\mathcal {O}) \ {\rm and }\ H^{s}(\partial \mathcal {O})$ are the Sobolev spaces, $n(x) \! \ne \! 0, x \! \in \! \overline{\mathcal {O}}$ is a complex C^∞-function and ν is the outward unit normal vector.

The problem (1)–(3) appears naturally when the scattering of plane waves is considered, and the inhomogeneity in $\mathbb {R}^d$ is located in $\mathcal {O}$ and is described by the index of refraction $\sqrt{n}$. We will be concerned with the cases d = 2, 3. Infinite differentiability of $\partial \mathcal {O}$ and n is assumed for the sake of simplicity; a finite smoothness is enough for all the results below. There are weaker definitions of ITEs when solutions (u, v) of the problem (1)–(3) are assumed to be only square integrable (boundary conditions (3) still can be defined since u and v satisfy the homogeneous elliptic equations). It will be shown in section 5 that these weak eigenfunctions of the ITE problem belong to the Sobolev space H² under conditions imposed in this paper, i.e., the a priori assumption $u,v\in H^2(\mathcal {O})$ does not reduce the set of ITEs.

We also consider the case when $\mathcal {O}$ contains a compact obstacle $\mathcal {V} \subset \mathcal {O}$, $\partial \mathcal {V} \in C^\infty$. In this case, equation (2) is replaced by

$$\begin{equation} {-}\Delta v - \lambda n(x)v =0, \quad x \in \mathcal {O} \backslash \mathcal {V}, \quad v\in H^2(\mathcal {O} \backslash \mathcal {V});\quad v(x)=0, \quad x \in \partial \mathcal {V}, \end{equation} \tag{ 4 }$$

while equation (1) remains valid in $\mathcal {O}$. For simplicity of notations, we will consider the problem (1)–(3) as a particular case of (1), (4), (3) with $\mathcal {V} = \emptyset$. The Dirichlet boundary condition on $\partial \mathcal {V}$ in (4) (as well as in our previous papers on ITEs) can be replaced by the Neumann or Robin boundary condition without any changes in the results or proofs.

Note that the problem (1)–(3) is neither elliptic nor formally symmetric, and therefore the properties of its spectrum cannot be obtained by soft arguments. The discreteness of the ITEs was proved first in [7] in the case when n is real and n(x) − 1 preserves the sign in the whole domain $\overline{\mathcal {O}}$. This result was extended in [3] to the case of domains with cavities, i.e., n(x) − 1 was allowed to vanish inside $\mathcal {O}$. In [23], it was proved that the set of the ITEs is discrete if $\mathcal {V} = \emptyset$ and n(x) − 1 does not vanish at the boundary $\partial \mathcal {O}$ (n can be complex valued). The case of piece-wise constant n was studied in [8]. In particular, it was shown there that the negative semi-axis does not contain ITEs when n ≠ 1 is a constant.

The index of refraction $\sqrt{n}$ is assumed to be real valued when the positive ITEs are studied. In [5], it was proved that the set of the positive ITEs is infinite if $\mathcal {V} = \emptyset$, and the function n(x) − 1 is not zero for all $x\in \overline{\mathcal {O}}$. Some Weyl-type lower estimates on the counting function for the positive ITEs were obtained in [21] in the case when n > 1 everywhere inside $\mathcal {O}$ (see also [12]). Note that n(x) = 1 at the boundary was allowed in [21], and the discreteness of the spectrum in this situation was justified there. The discreteness of ITEs and the existence of infinitely many positive ITEs were proved in [4] when $\mathcal {V} \ne \emptyset$ and n < 1 everywhere.

In [12], it is shown that in the case of $n(x)>1, x \in \overline{\mathcal {O}},$ all but finitely many complex transmission eigenvalues are confined to a parabolic neighborhood of the real positive semi-axis. In [13], the authors justified the completeness of the set of the interior transmission eigenfunctions under the same assumption on n(x).

In [14–16], we considered anisotropic problems and proved the discreteness of the ITEs, the existence of real ITEs and established the Weyl-type estimates for the real ITEs. Some of these results were known earlier (see the recent review [6]). We showed that, under weak assumptions, the anisotropic ITE problem is parameter-elliptic. This allowed us to broaden the scope of applications, simplify the proofs and obtain new results. However, our approach cannot be directly applied to the isotropic case that we consider here, since the isotropic problem is not elliptic. The extension of our previous results to the isotropic problems will be obtained in this paper by the reduction of the problem to an elliptic pseudo-differential operator (p.d.o.) (of a lower order) at the boundary using Dirichlet-to-Neumann maps for equations (1) and (4).

The isotropic problem (1), (4), (3) is considered in this paper when either

$$\begin{equation} n(x)-1 \ne 0, \quad x \in \partial \mathcal {O}, \end{equation} \tag{ 5 }$$

or

$$\begin{equation} n(x)-1 \equiv 0, \quad \frac{\partial }{\partial \nu } n(x) \ne 0, \quad x \in \partial \mathcal {O}. \end{equation} \tag{ 6 }$$

Let us stress again that these conditions on n are imposed only at the boundary of the domain. Condition (6) allows one to consider scattering problems in inhomogeneous media with a continuous index of refraction $\sqrt{n(x)}$. We also assume that there exists a closed sector $\Lambda \subset \mathbb {C}$ centered in the origin that does not contain any points of the following set $\mathcal {N}$:

$$\begin{equation} \mathcal {N} = \lbrace 1\rbrace \cup \left\lbrace \frac{1}{n(x)}, x \in \overline{ \mathcal {O}} \right\rbrace . \end{equation} \tag{ 7 }$$

If function $n(x)>0,x \in \overline{ \mathcal {O}},$ is real valued, then the latter assumption obviously holds for any sector that does not contain the positive semi-axis $\mathbb {R}^+$.

We will show that the set of ITEs is discrete with two possible accumulation points: zero and infinity, and zero is not an accumulation point if $\mathcal {V} = \emptyset$. Moreover, if $\mathcal {V} = \emptyset ,$ we show that there are at most finitely many ITEs in any closed sector $\Lambda \subset \mathbb {C}$ centered in the origin that does not contain any points of $\mathcal {N}$. The same is true if $\mathcal {V} \ne \emptyset$ and a neighborhood of the origin is cut off from Λ, i.e., Λ is replaced by $\Lambda \bigcap \lbrace |\lambda |>1\rbrace$.

We will also prove the existence of infinitely many real ITEs when $n(x)\!>\!0, \;\! x \!\in \!\overline{\mathcal {O}},$ is a real-valued function and

$$\begin{equation*} \sigma \left( Vol(\mathcal {O}) - \int _{\mathcal {O} \backslash \mathcal {V}} n^{d/2}(x)\,{\rm d}x \right)>0, \end{equation*}$$

where

$$\begin{equation} \sigma ={\rm sign}(n(x)-1),\quad x\in \mathcal {O},\quad {\rm dist}(x,\partial \mathcal {O})\ll 1, \end{equation} \tag{ 8 }$$

is the sign of n − 1 in a neighborhood of ∂Ω strictly inside of Ω. The constant σ is well defined due to the conditions imposed on n. Moreover, we will obtain a Weyl-type lower bound on the counting function N_T(λ) of the positive ITEs. Note that the condition imposed above holds, for example, in the following cases: (1) n(x) > 1 on $\partial \mathcal {O}$, but n(x) is small enough inside of $\mathcal {O}$, or (2) n(x) > 1 everywhere, but the obstacle $\mathcal {V}$ is large enough, or (3) n(x) < 1 on $\partial \mathcal {O}$, but n(x) is large enough inside of $\mathcal {O}$.

All the proofs are based on methods of elliptic p.d.o. Therefore, the assumptions on the smoothness of $\partial \mathcal {O}$ and n(x) are essential for us, while some of the earlier results mentioned above were proved for n(x) ∈ L^∞. Let us outline how methods of elliptic equations appear in the study of non-elliptic problem (1), (3), (4).

Let

$$\begin{equation} F(\lambda ), F_n(\lambda ):H^{3/2}(\partial \mathcal {O})\rightarrow H^{1/2}(\partial \mathcal {O}) \end{equation} \tag{ 9 }$$

be the Dirichlet-to-Neumann map for equations (1) and (4), respectively. Note that the domain of the operator F_n consists of functions defined on $\partial \mathcal {O}$, since the value v = 0 is fixed on $\partial \mathcal {V}$ (if $\mathcal {V}\ne \emptyset )$. Operators (9) are well defined when λ is not an eigenvalue of the Dirichlet problem for equations (1) and (4), respectively. In particular, from the Green formula, it immediately follows that these operators are well defined in any sector Λ that does not contain points of set $\mathcal {N}$.

Consider an arbitrary λ = λ₀ that is neither a pole of F(λ₀) nor F_n(λ₀). From (3), it follows that λ = λ₀ is an ITE if and only if the kernel of F_n(λ₀) − F(λ₀) is not empty. The situation when λ = λ₀ is a pole is quite similar and will be considered later. Operators (9) are elliptic p.d.o. of the first order. Their principal symbols do not depend on n, i.e., the principal symbol of the difference F_n − F is zero. However, this difference can be an elliptic operator of a lower order.

Our results are based essentially on old papers by Vainberg and Grushin [24, 25], who calculated the full symbols of different p.d.o. that map boundary values of one problem for an elliptic equation to the boundary values of another problem (their goal was to show that coercivity in some non-elliptic problems may occur due to the structure of the lower order terms of the full symbol of the operator). We will apply these calculations to prove the following two main lemmas.

Lemma 1.1. Let one of the conditions (5) or (6) hold. Let s = 1 if (5) holds, and s = 2 if (6) holds. Then, the following statements are valid.

• (1)  
  The difference

  $$\begin{equation} F_n(\lambda )-F(\lambda ):H^{\frac{3}{2}}(\partial \mathcal {O})\rightarrow H^{\frac{3}{2}+s}(\partial \mathcal {O}),\quad \lambda \ne 0, \end{equation} \tag{ 10 }$$

  depends meromorphically on λ and is an elliptic p.d.o. of order −s, when λ is not a pole of (10).If λ is not a pole of the operator F_n − F, and the map from global to local coordinates is orthogonal at the boundary ∂Ω, then the principal symbol of F_n(λ) − F(λ) in these local coordinates is

  $$\begin{eqnarray} &&(1-n(x))\frac{\lambda }{2|\xi |}, \quad x \in \partial \mathcal {O}, \quad \xi \in \mathbb {R}^{d-1},\quad { if}\ {(5)}\ {holds}, \end{eqnarray} \tag{ 11 }$$

  $$\begin{eqnarray} &&\frac{1}{4}\frac{\partial n(x)}{\partial \nu }\frac{\lambda }{|\xi |^{2}}, \quad x \in \partial \mathcal {O}, \quad \xi \in \mathbb {R}^{d-1},\quad {if}\ {(6)}\ {holds} . \end{eqnarray} \tag{ 12 }$$

  Each pole λ = λ₀ of this operator has order 1, and the residue $P_{\lambda _0}$ has an infinitely smooth integral kernel $P_{\lambda _0}(x,y)\in C^\infty$ in (x, y). The operator $P_{\lambda _0}$ is a projection on a finite-dimensional space spanned by the normal derivatives of the solutions of the homogeneous Dirichlet problems for equations (1) and (4). The regular part $F_n(\lambda )-F(\lambda )-\frac{P_{\lambda _0}}{\lambda -\lambda _0}$ of operator (10) is a p.d.o. of order −s, whose principal symbol is given by (11) and (12).
• (2)  
  If $\mathcal {V} = \emptyset$, then F_n(0) = F(0), and the operator

  $$\begin{equation*} G(\lambda ):=\frac{F_n(\lambda )-F(\lambda )}{\lambda }:H^{\frac{3}{2}}(\partial \mathcal {O})\rightarrow H^{\frac{3}{2}+s}(\partial \mathcal {O}),\quad |\lambda |\ll 1, \end{equation*}$$

  has a limiting value G(0). The operator G(λ) depends analytically on λ in a neighborhood of the origin and is Fredholm for each λ.

Remark 1. The upper index 3/2 in (10) can be replaced by any $m\in \mathbb {R}.$ We decided to use m = 3/2, since a part of the proof is slightly simpler in this case.

Remark 2. The operators F_n and F have order 1, and the difference F_n − F has order −1 or −2 (it is equal to −s), i.e., two or three first terms of the full symbol are canceled when the difference is taken.

The second lemma concerns the parameter-ellipticity of the operator λ⁻¹[F_n(λ) − F(λ)], which leads to the invertibility of this operator for large |λ|. We will formulate here the invertibility result.

Let $H^{m,k}(\partial \mathcal {O}),\;k>0,$ be the Hilbert space with the norm

$$\begin{equation} ||u||^2_{H^{m,k}(\partial \mathcal {O})}=||u||^2_{H^{m}(\partial \mathcal {O})}+k^{2m}||u||^2_{L^2(\partial \mathcal {O})}, \end{equation} \tag{ 13 }$$

where $H^{n}(\partial \mathcal {O})$ is the Sobolev space. Below, parameter k will be always equal to $\sqrt{|\lambda |}$. Let Λ be an arbitrary closed sector of the complex plane that does not contain the set $\mathcal {N}$ defined in (7), and let $\Lambda ^{\prime }=\Lambda \bigcap \lbrace |\lambda |>1\rbrace$. Recall that Λ does not contain poles of F_n and F.

Lemma 1.2. Let n(x) ≠ 1 on $\partial \mathcal {O}$ (i.e. (5) holds) and let sector Λ not contain any points of $\mathcal {N}$. Then, for each Λ' and each $m\in \mathbb {R}$, the operator

$$\begin{equation} \lambda ^{-1} [F_n(\lambda )-F(\lambda )]:H^{m,k}(\partial \mathcal {O})\rightarrow H^{m+1,k}(\partial \mathcal {O}),\quad \lambda \in \Lambda ^{\prime },\quad k=\sqrt{|\lambda |}, \end{equation} \tag{ 14 }$$

is uniformly bounded in λ. Moreover, there exists A = A(Λ') such that operator (14) is invertible when |k| > A and

$$\begin{equation} \| (F_n(\lambda )-F(\lambda ))^{-1}f\| _{H^{m,k}}\le C|\lambda |^{-1}\| f\| _{H^{m+1,k}}, \quad \lambda \in \Lambda ^{\prime }, \quad |k| > A. \end{equation} \tag{ 15 }$$

Let $n(x)\equiv 1, \frac{\partial n}{\partial \nu }\ne 1$ on $\partial \mathcal {O}$ (i.e. (6) holds) and let sector Λ not contain any points of $\mathcal {N}$. Then, for each Λ' and each $m\in \mathbb {R}$, the operator

$$\begin{equation} \lambda ^{-1}[F_n(\lambda )-F(\lambda )]:H^{m,k}(\partial \mathcal {O})\rightarrow H^{m+2,k}(\partial \mathcal {O}),\quad \lambda \in \Lambda ^{\prime },\quad k=\sqrt{|\lambda |}, \end{equation} \tag{ 16 }$$

is uniformly bounded in λ. Moreover, there exists A = A(Λ') such that operator (16) is invertible when |k| > A and

$$\begin{equation} \| (F_n(\lambda )-F(\lambda ))^{-1}f\| _{H^{m,k}}\le C|\lambda |^{-1}\| f\| _{H^{m+2,k}}, \quad \lambda \in \Lambda ^{\prime },\quad |k|>A. \end{equation} \tag{ 17 }$$

Lemmas 1.1 and 1.2 imply the following statement.

Theorem 1.3. Let one of the conditions (5) or (6) hold. Assume that there exists a closed sector $\Lambda \subset \mathbb {C}$ centered in the origin that does not contain any points of the set $\mathcal {N}$. Then the following statements hold.

• (1)  
  If $\mathcal {V} = \emptyset$, then the set of the ITEs for problem (1), (4), (3) is at most countable with the only possible accumulation point at infinity. Moreover, there is at most a finite number of the ITEs inside each closed sector Λ centered in the origin that does not contain points of $\mathcal {N}$. Thus, if n(x) is real valued at the boundary, then there is at most a finite number of the ITEs inside each closed sector Λ of complex λ-plane that does not contain the ray $\mathbb {R}^+$.
• (2)  
  If $\mathcal {V} \ne \emptyset$, then the set of the ITEs for problem (1), (4), (3) is at most countable with the only possible accumulation points at zero and infinity. Moreover, there is at most a finite number of the ITEs inside $\Lambda ^{\prime }=\Lambda \bigcap \lbrace |\lambda |>1\rbrace$.

Remark 3. If $\mathcal {V} = \emptyset$, then F_n(0) = F(0), and therefore λ = 0 is an ITE of infinite multiplicity. The multiplicities of all the other ITEs are finite due to lemma 1.1. If $\mathcal {V} \ne \emptyset$, then λ = 0 is not an ITE. The latter can be proved very easily for domains with the Lipshitz boundary and without assumptions on the smoothness of n. Indeed, assume that there exists a solution (u, v) of (1), (3), (4) for λ = 0. Denote $w=u-v \in H^2(\mathcal {O} \backslash \mathcal {V})$. Since $w =\frac{\partial w}{\partial \nu }\equiv 0$ on $\partial \mathcal {O}$, we have w ≡ 0 in $\mathcal {O} \backslash \mathcal {V}$. Thus, u ≡ v ≡ 0 on the boundary $\partial \mathcal {V}$. Therefore, u equals zero in $\mathcal {V}$, and therefore u ≡ 0 everywhere in $\mathcal {O}$.

Remark 4. The results above can be easily extended to the case when

$$\begin{equation*} \frac{\partial ^i (n(x)-1)}{\partial \nu ^i} \equiv 0,\quad i=0, \ldots ,m-1,\quad \frac{\partial ^m (n(x)-1)}{\partial \nu ^m} \ne 0, \quad x \in \partial \mathcal {O}. \end{equation*}$$

The following theorem is based on lemmas 1.1, 1.2 and the ideas developed in [16].

Theorem 1.4. Let $n(x),x\in \mathcal {O},$ be a real-valued function and let one of the assumptions (5) or (6) hold. Let

$$\begin{equation*} \gamma := \sigma \left( Vol(\mathcal {O}) -\int _{\mathcal {O} \backslash \mathcal {V}} n^{d/2}(x)\,{\rm d}x \right)>0, \end{equation*}$$

where σ is defined in (8).

Then, the set of positive ITEs is infinite, and moreover

$$\begin{equation*} N_T(\lambda ) \ge \frac{\omega _d}{(2\pi )^d} \gamma \lambda ^{d/2} + O(\lambda ^{(d-1)/2}), \quad \lambda \rightarrow \infty , \end{equation*}$$

where ω_d is the volume of the unit ball in $\mathbb {R}^d$ and N_T is the counting function of the positive ITEs.¹

Now we will prove theorems 1.3 and 1.4. In section 3, the results from [25] needed to prove the main lemmas will be reviewed. The lemmas will be proved in the last section.

Proof of theorem 1.3. Obviously, every λ = λ₀ that is not an eigenvalue of the Dirichlet problem for equation (1) or (4) is an ITE if and only if the kernel of F_n(λ₀) − F(λ₀) is not empty. Thus, theorem 1.3 will be proved if we justify the corresponding statements for the set of λ_i with a non-empty kernel of F_n(λ_i) − F(λ_i) instead of the set of the ITEs.

We will need the following extension [2] of the analytic Fredholm theorem: if a domain $\Omega \in \mathbb {C}$ is connected and an operator function T = T(λ), λ ∈ Ω, is finitely meromorphic and Fredholm, then the invertibility of T(λ) at one point λ ∈ Ω implies that the inverse operator function T⁻¹(λ), λ ∈ Ω, is finitely meromorphic and Fredholm. Recall that a meromorphic operator function T = T(λ): H₁ → H₂ in Hilbert spaces H_i is called finitely meromorphic if the principal part of the Laurent expansion at each pole λ = λ₀ is an operator of a finite rank (i.e., coefficients for negative powers of λ − λ₀ are finite-dimensional operators). This operator function is called Fredholm if the operator T(λ) is Fredholm at each regular point λ = λ₀, and the regular part of T(λ) is Fredholm at each pole of the function.

The first part of lemma 1.1 implies that the family of operators (10) is finitely meromorphic and Fredholm when $\lambda \in \mathbb {C}\backslash \ 0$. Lemma 1.2 guarantees the invertibility of (10) when $\lambda \in \Lambda \bigcap \lbrace |\lambda |>a\rbrace$ and a = a(Λ) is large enough. Thus, the above theorem on the inversion of the meromorphic family of operators can be applied, which leads to the discreteness of the ITEs in $\mathbb {C}\backslash \ 0$. In order to complete the proof of theorem 1.3, it remains only to show that λ = 0 cannot be a limiting point for the set of the ITEs if $\mathcal {V} = \emptyset$.

Assume that $\mathcal {V} = \emptyset$. Since operator (10) is invertible except possibly for a discrete set of points λ, the same is true for the operator G(λ) = λ⁻¹[F_n − F]. This and the second statement of lemma 1.1 allow us to apply the analytic Fredholm theorem to G(λ) in a neighborhood of the origin. Hence, G(λ) and therefore F_n − F = λG(λ) may have non-trivial kernels at most at finitely many points of this neighborhood. □

Proof of theorem 1.4. Recall that $n(x), x \in \mathcal {O},$ in the statement of the theorem is assumed to be a real-valued function.

Denote the set of the positive ITEs with their multiplicities taken into account by $\lbrace \lambda ^T_i\rbrace$. Similarly, denote the set of positive eigenvalues of the Dirichlet problem for −Δ in $\mathcal {O}$ by {λ_i}, and the set of positive λ > 0 for which equation (4) in $\mathcal {O} \backslash \mathcal {V}$ with the Dirichlet boundary condition at the boundary $\partial \left(\mathcal {O} \backslash \mathcal {V}\right)$ has a non-trivial solution by $\lbrace \lambda ^n_i\rbrace$. The corresponding counting functions will be denoted by

$$\begin{equation} N_T(\lambda )= \#\big\lbrace i:\alpha <\lambda ^T_i \le \lambda \big\rbrace , \quad N(\lambda )= \#\lbrace i:\lambda _i \le \lambda \rbrace , \quad N_n(\lambda )= \#\big\lbrace i:\lambda ^n_i \le \lambda \big\rbrace , \end{equation} \tag{ 18 }$$

where $\alpha \in \big(0,\min \big(\lambda _0,\lambda ^n_0\big)\big)$ is an arbitrary small enough positive number that does not belong to the set $\big\lbrace \lambda ^T_i\big\rbrace$. Note that we do not count positive ITEs on the segment [0, α] where we cannot guarantee that the number of ITEs is finite if $\mathcal {V}\ne \emptyset$.

We are going to prove the following estimate on N_T(λ) from below through the counting functions N(λ) and N_n(λ). □

Theorem 2.1. Let one of the assumptions (5) or (6) hold. Then, there exists a constant n⁻(1) ⩾ 0 such that

$$\begin{equation} N_T(\lambda ) \ge \sigma (N(\lambda )-N_n(\lambda ))-n^-(\alpha ), \quad \lambda >\alpha , \end{equation} \tag{ 19 }$$

where σ is defined in (8).

Remark. The constant n⁻(α) will be defined below during the proof.

Theorem 1.4 is a direct consequence of theorem 2.1 and the well-known (e.g., [20, theorem 1.2.1]) Weyl formula for N(λ) and N_n(λ). Hence, we need only to prove theorem 2.1.

Proof of theorem 2.1. An analogue of theorem 2.1 (and of theorem 1.4) for anisotropic media was proved in [16]. The proof was based on the ellipticity of the operator F_A − F, where A is the matrix that describes the anisotropy of the medium. Lemma 1.1 allows us to carry over all the arguments to the isotropic case. One of the main differences is that operator (10) is of negative order (the principal symbols of the terms in the difference get canceled), while the order of F_A − F is positive. Secondly, we will need to be more careful around the point λ = 0 where the ellipticity of (10) is lost. The proof of theorem 2.1 will be close to the one from [16]. We will provide a detailed proof below not only because of the differences mentioned above, but also because we were able to simplify some of the steps from [16].

We will assume first that

$$\begin{equation} \lbrace \lambda _i\rbrace \cap \lbrace \lambda ^n_i\rbrace = \emptyset . \end{equation} \tag{ 20 }$$

This case is more transparent. All additional details, needed to consider the general case (when (20) is violated), will be discussed at the very end of the proof. □

Step 1. OperatorB(λ) and its eigenvalues μ_j = μ_j(λ). We will say that a meromorphic operator function has a kernel at a pole (of the first order) if there is a non-empty intersection of the kernel of its residue with the kernel of its regular part. The dimension of this intersection will be called the dimension of the kernel of the operator.

From the definition of the ITEs and (20), it follows immediately that a point $\lambda =\lambda _0\in \mathbb {R}$ is an ITE if and only if the operator F(λ) − F_n(λ) has a non-trivial kernel at λ = λ₀. The multiplicity of the ITE coincides with the dimension of the kernel.

We will assume that (5) holds. Consider the operator

$$\begin{equation} B(\lambda ):=\sigma D (F(\lambda )-F_n(\lambda )) D \! : \! H^{3/2}(\partial \mathcal {O}) \rightarrow H^{1/2}(\partial \mathcal {O}), \qquad D=(1-\Delta _{\partial \mathcal {O}})^{1/2}, \end{equation} \tag{ 21 }$$

where $\Delta _{\partial \mathcal {O}}$ is the Laplace–Beltrami operator on $\partial \mathcal {O}$. If condition (6) holds instead of (5), then one needs only to replace D above by D^3/2 (alternatively, one can use D² and replace H^1/2 by H^−1/2). The dimensions of the kernels of operators B(λ) and F(λ) − F_n(λ) coincide, and therefore the following lemma is valid.

Lemma 2.2. Let (20) hold. Then, λ = λ₀ is an ITE if and only if the operator B(λ) has a non-empty kernel at λ = λ₀. The multiplicity of the ITE λ₀ is equal to the dimension of the kernel of B(λ₀).

We will use the operator B(λ) to count the number of the ITEs with their multiplicities taken into account. For this purpose, we are going to study the negative spectrum of the operator B(λ).

From the Green formulas for equations (1) and (4), it follows immediately that the operators F_n and F (and therefore, B(λ)) are symmetric when λ is real. By lemma 1.1, the operator B(λ) is an elliptic p.d.o. of order 1. Hence, if λ is not a pole of B, then the spectrum of B(λ) consists of a sequence {μ_j(λ)} of real eigenvalues of finite multiplicities, and

$$\begin{equation} |\mu _j(\lambda )|\rightarrow \infty \quad {\rm as} \quad j\rightarrow \infty . \end{equation} \tag{ 22 }$$

Note that the reason for introducing the operator D in (21) (which was not used in [16]) is to avoid considering the essential spectrum of the compact operator F(λ) − F_n(λ) at the point μ = 0.

The operator B = σD(F(λ) − F_n(λ))D has a positive principal symbol (see (11), (12) and (8)), and therefore (see [22, corollary 9.3]) it is bounded from below when λ is not a pole. Obviously, the bound can be chosen locally uniformly in λ, i.e., the following statement holds.

Lemma 2.3. For each closed interval $I \subset \mathbb {R}^+$ where the operator B(λ) is analytic, there exists a constant C = C(I) such that

$$\begin{equation*} \mu _j(\lambda )\ge -C, \quad \lambda \in I,\quad j=1,2,\ldots \end{equation*}$$

Lemma 2.4. If the operator B(λ) is analytic in a neighborhood of a point λ = λ₀, then all the eigenvalues μ = μ_j(λ) are analytic in this neighborhood.

If λ = λ₀ is a pole of the first order of the operator B(λ) and p is the rank of its residue P, then p eigenvalues μ = μ_j(λ) and their eigenfunctions have a pole at λ₀ and all the others are analytic in this neighborhood. The residues of the eigenvalues μ_j(λ) are the eigenvalues of the residue P of the operator B(λ).

Proof. The first statement is a well-known property of analytic self-adjoint operators (see [18, theorem XII.13]) whose spectrum consists of eigenvalues of finite multiplicities. In order to prove the second property, consider the operator A(λ) = (λ − λ₀)B(λ). It is analytic in a neighborhood of λ = λ₀ and has exactly p eigenvalues that do not vanish at λ₀. Let L_λ be the p-dimensional space spanned by the corresponding eigenfunctions of the operator A(λ). L_λ is analytic in a neighborhood of λ₀ due to the above-mentioned property of analytic self-adjoint operators. By using L_λ and its orthogonal complements, one can write the original operator B(λ) in a neighborhood of λ₀ in a block form, where the block that corresponds to L_λ has a pole and the second block is analytic. After that, the statements of the second part of the lemma become obvious. □

Step 2. Relation between the set of the ITEs $\big\lbrace \lambda _i^T\big\rbrace$ and the eigenvalues μ_j = μ_j(λ). Denote by $n^-(\lambda ), \lambda \notin \lbrace \lambda _i\rbrace \bigcup \big\lbrace \lambda ^n_i\big\rbrace \bigcup \big\lbrace \lambda ^T_i\big\rbrace$, the number of negative eigenvalues μ_j(λ) of the operator B(λ). From (22) and lemmas 2.3 and 2.4, it follows that this number is finite for each λ.

Let us evaluate the difference n⁻(λ') − n⁻(α) by moving λ from λ = α to λ = λ' > α. Here, α > 0 is the constant defined in (18). The eigenvalues μ_j(λ) are meromorphic functions of λ, and the number of negative eigenvalues μ_j(λ) < 0 changes only when some of them pass through the 'edges' of the interval $\mathbb {R}^-_\mu =(-\infty ,0)$. Denote by n₁(λ') the change in n⁻(λ') − n⁻(α) due to the eigenvalues going through μ = −∞ when λ moves from α to λ'. Similarly, denote by n₂(λ') the change in n⁻(λ') − n⁻(α) due to the eigenvalues going through μ = 0 when λ moves from α to λ'. Then,

$$\begin{equation} n^-(\lambda ^{\prime })-n^-(\alpha )=n_1(\lambda ^{\prime })+n_2(\lambda ^{\prime }). \end{equation} \tag{ 23 }$$

Note that the annihilation or the birth of μ_j(λ) at μ = −∞ may occur only when λ passes through a pole λ = λ₀ of an eigenvalue μ_j(λ). Let us denote by δn₁(λ₀) the jump of n₁ at a pole λ = λ₀ of the operator B(λ) due to some of μ_j going through negative infinity.

Lemma 2.5. The following relation holds for every pole λ = λ₀ > 0 of the operator B(λ):

$$\begin{equation} \delta n_1(\lambda _0)=s^--s^+, \end{equation} \tag{ 24 }$$

where s⁻ and s⁺ are the numbers of negative and, respectively, positive eigenvalues of the residue $P_{\lambda _0}$ of the operator B(λ). When (20) holds, the latter relation becomes

$$\begin{equation} \delta n_1(\lambda _0)=\sigma (m_n-m_0), \end{equation} \tag{ 25 }$$

where m_n and m₀ are ranks of the residues of the operators F_n(λ) and F(λ), respectively, at the pole.

Proof. Recall that the eigenvalue μ_j(λ) may have poles only of the first order (see lemma 1.1). If an eigenvalue μ = μ_j(λ) has a pole at λ = λ₀ > 0 with a positive residue, then $\lim _{\lambda \rightarrow \lambda _0\pm 0}\mu _j(\lambda )=\pm \infty$, and therefore μ_j(λ) leaves the negative semi-axis μ < 0 through −∞ when $\lambda \rightarrow \lambda _0^-$. Similarly, if the residue is negative, then the eigenvalues enter the semi-axis when $\lambda = \lambda _0^+$. This proves (24).

It is enough to prove (25) when F_n(λ) has a pole at λ = λ₀. The case when F(λ) has a pole is similar. Thus, we assume that

$$\begin{equation} F_n(\lambda )=\frac{P_{\lambda _0}}{\lambda -\lambda _0} + F^0_n(\lambda ), \end{equation} \tag{ 26 }$$

where $P_{\lambda _0}$ is a finite-dimensional (of rank m_n) operator and $F_n^0(\lambda )$ is a smooth operator in a neighborhood of λ₀. The operator $P_{\lambda _0}$ has exactly m_n non-zero eigenvalues (their number coincides with the rank). We will show below that all of them are positive. Then, the residue $P=-\sigma DP_{\lambda _0}D$ of operator B(λ) also has exactly m_n non-zero eigenvalues and their signs coincide with the sign of −σ. Hence, (25) follows from (24) and the last statement of lemma 2.4. It remains to show that $P_{\lambda _0}\ge 0$. The latter is an obvious consequence of (26) and the following important statement [11]: the operator $\frac{\rm d}{{\rm d}\lambda }F_n(\lambda )$ is negative at every λ that is not a pole of the operator. The proof of lemma 2.5 is complete, but we will recall the proof of the statement from [11] to have all the details readily available to the reader.

Let u = u(λ) be the solution of the equation Δu + λn(x)u = 0 in $\mathcal {O} \backslash \mathcal {V}$ with the Dirichlet data φ at the boundary $\partial \mathcal {O}$ and let u = 0 on $\partial \mathcal {V}$. Its derivative $u^{\prime }=\frac{{\rm d} u(\lambda )}{{\rm d} \lambda }$ satisfies the equation

$$\begin{equation*} (\Delta + \lambda n) u ^{\prime } + n u = 0. \end{equation*}$$

Let us multiply this equation by $\overline{u}$, integrate over $\mathcal {O} \backslash \mathcal {V}$ and apply Green's formula. Since u' = 0 on the boundary, we obtain that

$$\begin{equation*} \int _{\partial \mathcal {O}} \frac{\partial u ^{\prime }}{\partial \nu } \overline{u}\,{\rm d}S + \int _{\mathcal {O} \backslash \mathcal {V}}n(x) |u|^2\, {\rm d}x=0, \end{equation*}$$

which can be rewritten as

$$\begin{equation*} \frac{\rm d}{{\rm d}\lambda } (F_n(\lambda ) \varphi ,\varphi )= -\int _{\mathcal {O} \backslash \mathcal {V}}n(x) |u|^2\, {\rm d}x<0. \end{equation*}$$

Thus, $\frac{\rm d}{{\rm d}\lambda }F_n(\lambda )<0$. □

By summation of inequalities (25) over all the poles λ₀ on the interval (α, λ), we obtain the following relation:

$$\begin{equation} n_1(\lambda ) = \sigma (N_n(\lambda )-N(\lambda )),\quad \lambda >\alpha , \end{equation} \tag{ 27 }$$

where N_n and N are the counting functions defined in (18). From (23) and (27), it follows that

$$\begin{equation} n^-(\lambda )-n^-(\alpha )+\sigma (N(\lambda )-N_n(\lambda )) = n_2(\lambda ),\quad \lambda >\alpha . \end{equation} \tag{ 28 }$$

Due to lemma 2.2, the value of the counting function N_T(λ) for the ITEs is equal to the number of zero values of all the eigenvalues μ_j(τ), j = 1, 2, ..., when τ belongs to the interval (α, λ). The function n₂(λ) also counts the number of zero values of μ_j(τ), j = 1, 2, ..., when τ changes from α to λ, but n₂(λ) counts these zeroes with coefficients ±1 or 0. The choice of this coefficient depends on whether the corresponding μ_j(τ) enters the semi-axis $\mathbb {R}_\mu ^-$, exits it or does not change location with respect to the semi-axis when τ changes from α to λ and passes through the point where μ_j(τ) = 0. Thus, N_T(λ) ⩾ n₂(λ). This and (28) justify (19) since n⁻(λ) ⩾ 0.

Step 3. The case when (20) is violated. We have additional ITEs in this case, and the following statement (which is also an immediate consequence of the definition of the ITEs) replaces lemma 2.2.

Lemma 2.6. A point λ = λ₀ is an ITE if and only if the operator B(λ₀) has a non-trivial kernel or the following two conditions hold.

• (1)  
  λ = λ₀ is an eigenvalue of the Dirichlet problem for −Δ and for equation (4), i.e., λ = λ₀ is a pole for both F(λ) and F_n(λ).
• (2)  
  The ranges of the residues of the operators F(λ) and F_A(λ) at the pole λ = λ₀ have a non-trivial intersection.

Moreover, the multiplicity of the ITE λ = λ₀ in all cases is equal to m₁ + m, where m₁ is the dimension of the kernel of the operator B(λ₀) and m is the dimension of the intersection of the ranges of the residues of the operators F(λ) and F_n(λ) at the pole λ = λ₀ (m = 0 if λ = λ₀ is not a pole).

Equality (25) must by replaced now by the following inequality, which is valid for every pole λ = λ₀ > 0 of the operator B(λ):

$$\begin{equation} |\delta n_1(\lambda _0)-\sigma (m_n-m_0)|\le m. \end{equation} \tag{ 29 }$$

Indeed, (24) remains valid in our case, but now we cannot find s^± explicitly. However, it is not difficult to show that |(s⁻ − s⁺) − σ(m_n − m₀)| ⩽ m, which leads to (29). The latter inequality follows easily from the fact (which can be found in the proof of lemma 2.5) that −σP ⩾ 0 on the space $DV_n^\perp$ and −σP ⩽ 0 on the space $DV_0^\perp$. Here, D is the operator defined in (21), V_n and V₀ are the ranges of the residues of F_n and F, respectively, and $V_n^\perp , V_0^\perp$ are the subspaces of the elements in V_n, V₀ that are orthogonal to $V_n\bigcap V_0$. If more details are needed, they can be found in [16, lemma 2.4].

We will call λ = λ₀ a singular ITE, and m will be called its multiplicity, if the last two conditions of lemma 2.6 hold. Let us denote by R(λ) the counting function of the singular ITEs (the number, with multiplicities taken into account, of the singular ITEs whose values do not exceed λ).

By summation of inequalities (29) over all the poles λ₀ on the interval (α, λ), we obtain the following analogue of (27):

$$\begin{equation} |n_1(\lambda )-\sigma (N_n(\lambda )-N(\lambda ))|\le R(\lambda ), \end{equation} \tag{ 30 }$$

which leads to the following analogue of (28):

$$\begin{equation} n^-(\lambda )-n^-(\alpha )+\sigma (N(\lambda )-N_n(\lambda ))\le R(\lambda )+n_2(\lambda ). \end{equation} \tag{ 31 }$$

It remains to note that N_T(λ) ⩾ n₂(λ) when only non-singular ITEs are counted (the non-singular ITEs are related to the non-trivial kernels of B(λ) and can be compared to n₂(λ)), and therefore, N_T(λ) ⩾ n₂(λ) + R(λ) if all the ITEs are counted. This and (31) imply (19).

Consider the problem

$$\begin{equation} \left\lbrace \begin{array}{@{}ll@{}} Au=0,& x \in \mathcal {O}, \\ u=\varphi , & x \in \partial \mathcal {O}, \end{array} \right. \end{equation} \tag{ 32 }$$

where A is an elliptic differential operator of the second order with infinitely smooth coefficients, and the boundary $\partial \mathcal {O}$ is also infinitely smooth.

Theorem 3.1. Let problem (32) be uniquely solvable. Then, the operator $F:\varphi \rightarrow \frac{\partial u}{\partial \nu }$ is an elliptic p.d.o. on $\partial \mathcal {O}$ of order 1 and its full symbol can be easily found by the procedure described below.

This theorem is a particular case of a more general statement proved in [25]. An elliptic system A of an arbitrary order is considered there with two different boundary operators B₁ and B₂ such that each of them complements A to an elliptic boundary value problem. It is proved there that the operator F: B₁u → B₂u is an elliptic p.d.o. on $\partial \mathcal {O}$, and the full symbol of this operator is calculated. In particular, the results of [25] imply that the full symbol of the D-to-N operator F has the following form (these calculations can be also found in the later publications [9, chapter VII, 17]).

Let V be a small neighborhood of a point on $\partial \mathcal {O}$ with local coordinates (y₁, ..., y_{d − 1}, t) such that $\partial \mathcal {O}$ in V is given by the equation t = 0, and $V\bigcap \mathcal {O}$ is defined by |y|² + t² < ε², t > 0. Let $A(y,t,i\frac{\partial }{\partial y},i\frac{\partial }{\partial t})$ be the operator A rewritten in local coordinates (y, t). The symbol of this operator in new coordinates is A(y, t, ξ, τ). A function Φ(z, t, ξ, τ) will be called the generalized homogeneous of order γ if

$$\begin{equation*} \Phi (\kappa z,\kappa t,\kappa \xi , \kappa \tau ) = \kappa ^{\gamma }\Phi (z,t,\xi ,\tau ) \end{equation*}$$

for every κ > 0. For each $x\in V\bigcap \partial \mathcal {O}$ and N > 0, the symbol of A can be written in the form

$$\begin{equation} A(y,t,\xi ,\tau ) = \sum _{j=0}^N A_j(x,y-y(x),t,\xi ,\tau )+ A_N^{\prime }(x,y-y(x),t,\xi ,\tau ), \end{equation} \tag{ 33 }$$

where A_j(x, z, t, ξ, τ) are generalized homogeneous polynomials in (z, t, ξ, τ) of order 2 − j and

$$\begin{equation*} |A_N^{\prime }(x,\kappa z,\kappa t,\kappa \xi , \kappa \tau )|\le C(x,z,t,\xi , \tau )\kappa ^{1-N} \quad {\rm when}\quad |z|^2+t^2<1 ,\quad \kappa \rightarrow 0. \end{equation*}$$

In order to obtain expansion (33), one needs to write A as a Taylor series in z and t centered at the point (y(x), 0) and group together the terms of the same order.

Let

$$\begin{eqnarray} &&\widetilde{A}_j = A_j\left(x,-{\rm i}\frac{\partial }{\partial \xi },t,\xi ,{\rm i}\frac{\partial }{\partial t}\right). \end{eqnarray} \tag{ 34 }$$

Consider the following recursive system of ODEs on the half-line t > 0 (which depend on the parameters x and ξ):

$$\begin{eqnarray} &&\widetilde{A}_0\, E_0 (x,t,\xi ) =0, \end{eqnarray} \tag{ 35 }$$

$$\begin{eqnarray} &&\widetilde{A}_0\, E_1 (x,t,\xi ) = -\widetilde{A}_1 E_0 (x,t,\xi ), \end{eqnarray} \tag{ 36 }$$

$$\begin{eqnarray} &&\widetilde{A}_0\, E_2 (x,t,\xi )= -\widetilde{A}_1 E_1 (x,t,\xi ) - \widetilde{A}_2 E_0 (x,t,\xi ), \end{eqnarray} \tag{ 37 }$$

$$\begin{eqnarray} &&\qquad\qquad\,\ldots \nonumber \\ &&\widetilde{A}_0 E_j (x,t,\xi )= -\widetilde{A}_1 E_{j-1} (x,t,\xi ) -\ldots - \widetilde{A}_j E_0 (x,t,\xi ) . \end{eqnarray} \tag{ 38 }$$

From the ellipticity of the operator A, it follows that this system has a unique solution {E_i}, i = 1, 2, ..., in the class of functions that decay at infinity and satisfy the following initial data:

$$\begin{equation*} E_0(x,t,\xi )|_{t=0}=1, \qquad E_j(x,t,\xi )|_{t=0}=0,\quad j>0. \end{equation*}$$

The full symbol F(x, ξ) of F is given by the following asymptotic series [25, theorem 14]:

$$\begin{equation} F(x,\xi )=-\sum _{j=0}^\infty \left( \frac{\rm d}{{\rm d} t} E_{j}\right)|_{t=0} . \end{equation} \tag{ 39 }$$

Example. As an example, consider A = −Δ + n(x)λ. Then, A₀ = A₀(x, 0, 0, ξ, τ) is a homogeneous polynomial of the second order, which has exactly one root in τ such that Imτ > 0 for ξ ≠ 0. Denote this root by $\tau =\tau (x,\dot{\xi })|\xi |, \dot{\xi }=\frac{\xi }{|\xi |}.$ Hence, $E_0={\rm e}^{{\rm i}t\tau (x,\dot{\xi })|\xi |}$, and the principal symbol of the D-to-N operator F in the chosen local coordinates is $-i\tau (x,\dot{\xi })|\xi |$. If the transformation matrix from global to local coordinates is orthogonal on the boundary $\partial \mathcal {O}$, then the Laplacian in the local coordinates is preserved on the boundary. Thus, $A_0=\tau ^2+|\xi |^2,\;\widetilde{A}_0=-\frac{{\rm d}^2}{{\rm d}t^2}+|\xi |^2$ and $\tau (x,\dot{\xi })=i$. Therefore, E₀ = e^−t|ξ|, and the principal symbol of operator F is |ξ| in these local coordinates.

Proof of lemma 1.1. It is easy to use the results of the previous section and obtain formulas (11) and (12) for the principal symbol of F_n − F. It will take much longer to justify the analytic properties of this operator in specific spaces indicated in (10).

Step 1. Proof of the first part of the lemma when λ belongs to a disc |λ − λ₀| < athat is free of eigenvalues of both Dirichlet problems: for equations (1) and (4). Theorem 3.1 and calculations of the symbol from the previous section can be applied in this case.

If A = −Δ + n(x)λ, then we denote operators $\widetilde{A}_k$ and functions E_k introduced in the previous section by $\widetilde{A}^n_k$ and $E_k^n$, respectively, and we preserve the previous notations (without index n) if n ≡ 1. Obviously, $\widetilde{A}_0^n=\widetilde{A}_0, \widetilde{A}_1^n=\widetilde{A}_1$ and $\widetilde{A}_2^n-\widetilde{A}_2=\lambda (1-n(x))$. For the sake of transparency of the proof, we will assume that $\widetilde{A}_0=\widetilde{A}^n_0=-\frac{{\rm d}^2}{{\rm d}t^2}+|\xi |^2$, see the example above (all the calculations could be easily made in arbitrary local coordinates). Hence, $E_0^n=E_0={\rm e}^{-t|\xi |}, E_1^n=E_1$, and

$$\begin{equation} \left[-\frac{{\rm d}^2}{{\rm d}t^2}+|\xi |^2\right]\big(E_2^n-E_2\big)=\lambda (n-1)\, {\rm e}^{-t|\xi |}. \end{equation} \tag{ 40 }$$

The function

$$\begin{equation*} Y(t)=\frac{\lambda (1-n(x))}{2|\xi |^2}t\,{\rm e}^{-t|\xi |} \end{equation*}$$

is a particular solution of equation (40). It vanishes at t = 0 and at infinity, i.e., $E_2^n-E_2=Y(t)$. Thus, the first two terms of the full symbol of the operator F_n − F are zeros, and the next one is equal to $-\frac{\rm d}{{\rm d}t}Y(t)|_{t=0}$. The latter expression coincides with (11). Hence, F_n − F is a p.d.o. of the order −1 with the principal symbol (11).

Let now $n(x)= 1, \frac{\partial n}{\partial \nu }\ne 0$ at $\partial \mathcal {O}$. In this case, $\widetilde{A}_j^n=\widetilde{A}_j, j=0,1,2,$ and $\widetilde{A}_3^n-\widetilde{A}_3=t\lambda \frac{\partial n}{\partial \nu }$ (note that t and ν have different directions). Hence, $E_j^n=E_j, j=0,1,2$, and the following equation holds for $E_3^n-E_3$:

$$\begin{equation} \left[-\frac{{\rm d}^2}{{\rm d}t^2}+|\xi |^2\right]\big(E_3^n-E_3\big)=-t\lambda \frac{\partial n}{\partial \nu }\, {\rm e}^{-t|\xi |}. \end{equation} \tag{ 41 }$$

The solution of (41) that decays at infinity and vanishes at t = 0 has the form

$$\begin{equation} E_3^n-E_3=-\lambda \frac{\partial n}{\partial \nu }\left(\frac{t^2}{4|\xi |}+\frac{t}{4|\xi |^2}\right)\,{\rm e}^{-t|\xi |}. \end{equation} \tag{ 42 }$$

Hence, the first non-zero term of the full symbol of the operator F_n − F is given by (12).

We proved (10)–(12) for λ in a disc |λ − λ₀| < a that does not contain poles of F_n and F. Now we are going to study the analytic properties of operator (10). We will do it when $n(x)= 1, \frac{\partial n}{\partial \nu }\ne 0$ at $\partial \mathcal {O}$. The case of $n(x)\ne 1, x\in \partial \mathcal {O},$ can be studied similarly. One of the difficulties in the proof of the analyticity of operator (10) is related to the fact that the range of operators F_n and F (i.e. space $H^{1/2}(\partial \mathcal {O}))$ is wider than the range of the difference of these operators indicated in (10). In order to prove the analyticity of operator (10), we will use the Taylor expansion of F_n − F at λ = 0 since the coefficients λ^j, j > 0, in the Taylor expansion have better smoothing properties than the operator itself. Indeed, let us write the solution of the problem (4) with the Dirichlet condition $v|_{\partial \mathcal {O}}=\phi$ in the form

$$\begin{equation} v=v_0(x)+\lambda v_1(x)+ w(\lambda ,x), \end{equation} \tag{ 43 }$$

where v₀ and v₁ do not depend on λ and

$$\begin{equation} {-}\Delta v_0=0,\quad x\in \mathcal {O}\backslash \mathcal {V},\quad v_0|_{\partial \mathcal {V}}=0,\quad v_0|_{\partial \mathcal {O}}=\phi ; \end{equation} \tag{ 44 }$$

$$\begin{equation} {-}\Delta v_1=n(x)v_0,\quad x\in \mathcal {O}\backslash \mathcal {V},\quad v_1|_{\partial \mathcal {V}}=v_1|_{\partial \mathcal {O}}=0; \end{equation} \tag{ 45 }$$

$$\begin{equation} {-}\Delta w-\lambda n(x)w=\lambda ^2 n(x)v_1,\quad x\in \mathcal {O}\backslash \mathcal {V},\quad w|_{\partial \mathcal {V}}=w|_{\partial \mathcal {O}}=0. \end{equation} \tag{ 46 }$$

Then,

$$\begin{equation*} \Vert v_0\Vert _{H^2}\le C \Vert \phi \Vert _{H^{3/2}(\partial \mathcal {O})},\quad \Vert v_1\Vert _{H^4}\le C \Vert v_0\Vert _{H^2}\le C\Vert \phi \Vert _{H^{3/2}(\partial \mathcal {O})},\quad \Vert w\Vert _{H^6}\le C \Vert \phi \Vert _{H^{3/2}(\partial \mathcal {O})}, \end{equation*}$$

and the operator $\Gamma :H^{3/2}(\partial \mathcal {O})\rightarrow H^6(\mathcal {O}\backslash \mathcal {V})$ that maps ϕ into w is bounded and analytic in λ when |λ − λ₀| < a. By taking the normal derivative on $\partial \mathcal {O}$ on both sides of (43), we obtain the following representation for F_n(λ):

$$\begin{equation*} F_n(\lambda )=F_n^0+\lambda F_n^1+ F_n^2(\lambda ),\quad F_n^2(\lambda ):H^{3/2}(\partial \mathcal {O})\rightarrow H^{9/2}(\partial \mathcal {O}), \quad |\lambda -\lambda _0|<a, \end{equation*}$$

where operators $F_n^0$ and $F_n^1$ do not depend on λ, and $F_n^2$ depends on λ analytically. A similar representation is valid for F(λ). Hence,

$$\begin{eqnarray} &&F_n(\lambda )-F(\lambda )=G_n^0+\lambda G_n^1+G_n^2(\lambda ),\quad G_n^2(\lambda ):H^{3/2}(\partial \mathcal {O})\rightarrow H^{9/2}(\partial \mathcal {O}), \quad |\lambda -\lambda _0|<a, \nonumber\\ \end{eqnarray} \tag{ 47 }$$

where the operators $G_n^0$ and $G_n^1$ do not depend on λ, and $G_n^2$ is analytic in λ.

Since the operator $F_n-F:H^{3/2}(\partial \mathcal {O})\rightarrow H^{7/2}(\partial \mathcal {O})$ is bounded for each λ (it is a p.d.o. of order −2), from (47) it follows that operators $G_n^0 \ {\rm and }\ G_n^1$ are bounded in the same spaces. Thus, (47) implies that operator (10) is analytic when |λ − λ₀| < a.

Step 2. Proof of the first part of lemma when λ₀ is a pole of either F_n or F (or both operators). We could repeat the previous arguments, but we will need to apply the resolvent ( − Δ − λn(x))⁻¹ to both sides of (46) in order to obtain w. Thus, the operator $G_n^2$ will have a pole of the first order at λ = λ₀, whose residue can be expressed through the residue of the resolvent. This approach leads to a slightly weaker result than the one stated in the lemma: it gives the description of the residue, but the regular part of F_n − F will be represented as a p.d.o. plus a smoother operator. While this result is sufficient for all applications in this paper, we decided to spend a little more time and prove the exact statement of lemma 1.1, i.e., to prove that the regular part is a p.d.o. up to an infinitely smoothing operator, and the principal symbol is given by (11) and (12).

In order to study the case when λ₀ is a pole, we perturb equations (1) and (4) by adding an infinitely smooth term q(x) ∈ C^∞ to the potential n(x) in these equations. For example, equation (1) will now take the form −Δu − λ(1 + q(x))u = 0. The goal of these perturbations is to get rid of the eigenvalue at λ = λ₀, while preserving the symbols of the regular parts of operators F_n and F. We may need to choose different potentials for these two equations, but we will use the same notation q for both equations (we never compare potential terms below).

We choose terms q that satisfy the following two requirements: q vanish in a neighborhood of $\partial \mathcal {O}$, and the homogeneous Dirichlet problems for both equations (1) and (4) with the terms q added have only trivial solutions when λ = λ₀. For example, in order to achieve the second requirement in the case of real-valued n(x), one can take q(x) = iq₁(x), q₁(x) ⩾ 0. If n(x) is complex valued, then one can take q(x) = −α(x)n(x), where α ∈ C^∞, 0 ⩽ α ⩽ 1, α = 1 outside of a small enough neighborhood of $\partial \mathcal {O}, \alpha$ vanishes in a smaller neighborhood of $\partial \mathcal {O}$.

Since λ = λ₀ is not an eigenvalue for F_n or F, there exists an a > 0 such that the disc |λ − λ₀| < a does not contain eigenvalues of the Dirichlet problem for equations (1) or (4) with the additional term q added. Thus, the D-to-N operators F_{n, q}, F_q, |λ − λ₀| < a, are defined for these equations.

Operators F_{n, q}, F_q, |λ − λ₀| < a, can be studied absolutely similarly to the operators with q = 0. They are p.d.o. of the first order. Their full symbols can be constructed exactly as for operators with q = 0. Moreover, since q = 0 in a neighborhood of $\partial \mathcal {O}$, the construction of the full symbol of these operators does not depend on q at all. One can also repeat the arguments leading to (47) and obtain a similar representation for F_{n, q} − F_q when $\lambda \in \mathbb {R}^+$. We will need only to add the term q to n in equations (44)–(46). Hence,

$$\begin{equation} F_{n,q}(\lambda )-F_q(\lambda ):H^{\frac{3}{2}}(\partial \mathcal {O})\rightarrow H^{\frac{3}{2}+s}(\partial \mathcal {O}),\quad |\lambda -\lambda _ 0|<a, \end{equation} \tag{ 48 }$$

is an elliptic p.d.o. of order −s, which is analytic in λ, and its principal symbol is given by (11) or (12).

Note that operator (48) differs from (10) by I₁ + I₂, where I₁ = F_{n, q}(λ) − F_n(λ), I₂ = F_q(λ) − F(λ). We are going to show that

$$\begin{equation} I_1=F_{n,q}(\lambda )-F_n(\lambda )=\frac{P_{\lambda _0,n}}{\lambda -\lambda _0}+Q_n(\lambda ),\quad |\lambda -\lambda _ 0|<a, \end{equation} \tag{ 49 }$$

where integral kernels $P_{\lambda _0,n}(x,y), Q_n(\lambda ,x,y), x,y\in \partial \mathcal {O},$ of operators $P_{\lambda _0,n}$, Q_n(λ) are infinitely smooth functions of their arguments, $P_{\lambda _0,n}$ does not depend on λ, Q(λ, x, y) depends on λ analytically, and $P_{\lambda _0,n}$ is a projection on the space spanned by the normal derivatives of the solutions of the homogeneous Dirichlet problem for equation (4). Thus, I₁ has a pole, but its principal part Q_n is an infinitely smoothing operator. The validity of (49) for arbitrary n implies, in particular, its validity for I₂, where n ≡ 1. Hence, (48) and (49) together justify the first statement of lemma 1.1. It remains only to prove (49).

We have

$$\begin{equation*} F_{n}(\lambda )\phi =\frac{\partial }{\partial \nu }v|_{\partial \mathcal {O}},\qquad F_{n,q}(\lambda )\phi =\frac{\partial }{\partial \nu }v_1|_{\partial \mathcal {O}}, \end{equation*}$$

where v, v₁ are the solutions of the following problems (for shortness, we will assume that $\mathcal {V}=\emptyset$, but one can easily add $\mathcal {V}$ below)

$$\begin{eqnarray} &&{-}\Delta v-\lambda n(x) v=0,\quad x\in \mathcal {O},\quad v|_{\partial \mathcal {O}}=\phi ,\\ &&{-}\Delta v_1-\lambda (n(x)+q(x)) v_1=0,\quad x\in \mathcal {O},\quad v_1|_{\partial \mathcal {O}}=\phi . \end{eqnarray} \tag{ 50 }$$

Thus, $(F_{n,q}- F_{n})\phi =\frac{\partial }{\partial \nu }w|_{\partial \mathcal {O}}$, where w satisfies

$$\begin{equation} {-}\Delta w-\lambda n(x)w=\lambda q(x)v_1,\quad x\in \mathcal {O},\quad w|_{\partial \mathcal {O}}=0. \end{equation} \tag{ 51 }$$

Hence,

$$\begin{equation*} (F_{n,q}- F_{n})\phi =\frac{\partial }{\partial \nu }\int _{\mathcal {O}}R_\lambda (x,y)\lambda qv_1(y)\,{\rm d}y|_{\partial \mathcal {O}}, \end{equation*}$$

where R_λ(x, y) is the kernel of the resolvent ( − Δw − λn(x))⁻¹ (for the operator with zero Dirichlet condition). The resolvent has a pole of the first order at λ = λ₀.

Since problem (50) does not have eigenvalues in the disc |λ − λ₀| < a and q = 0 in a neighborhood of $\partial \mathcal {O}$, it follows that

$$\begin{equation*} q(x)v_1=\int _{\partial \mathcal {O}}K(\lambda ,x,y)\phi (y)\,{\rm d}s_y, \end{equation*}$$

where the kernel K is infinitely smooth and analytic in λ. The latter two formulas imply (49) since in both formulas x and y are separated. The first statement of lemma 1.1 is proved.

Step 3. Proof of the second part of the lemma. Let $\mathcal {V}=\emptyset$. We note that (47) remains valid when λ₀ = 0. In particular, $G_n^2(\lambda )$ is analytic at λ = 0 in this case. Further, $G_n^0=0$ when $\mathcal {V}=\emptyset$ since functions v₀ determined by (44) are the same for both operators F_n and F. Thus, (47) takes the form

$$\begin{equation} F_n(\lambda )-F(\lambda )=\lambda G_n^1+G_n^2(\lambda ),\quad G_n^2(\lambda ):H^{3/2}(\partial \mathcal {O})\rightarrow H^{9/2}(\partial \mathcal {O}), \quad |\lambda |<a\ll 1, \end{equation} \tag{ 52 }$$

where operator $G_n^1$ does not depend on λ, and $G_n^2$ is analytic in λ.

It is an obvious consequence of the definition of operators F_n and F that the operators F_n(0) and F(0) coincide. Thus, $G_n^2(0)=0$, and (52) implies that the operator $G(\lambda )=\frac{F_n(\lambda )-F(\lambda )}{\lambda }$ can be extended analytically at λ = 0. It remains only to show that G(0) is Fredholm.

It is proved in the first part of the lemma that the operator F_n(λ) − F(λ) is an elliptic p.d.o. of order −s when λ ≠ 0 and is not a pole. Thus, the operator

$$\begin{equation} \lambda G_n^1+G_n^2(\lambda ):H^{3/2}(\partial \mathcal {O})\rightarrow H^{3/2+s}(\partial \mathcal {O}),\quad 0<|\lambda |<a, \end{equation} \tag{ 53 }$$

is Fredholm. Since s = 1 or 2, from (52) and the Sobolev imbedding theorem it follows that the second term on the left-hand side of (53) is a compact operator (by the same reason, $(G^2)^{\prime }:=\lim _{\lambda \rightarrow 0}\lambda ^{-1}G_n^2(\lambda )$ is compact). Thus, $G_n^1$ is Fredholm. Since (G²)' is Fredholm, the operator $G(0)=G_n^1+(G^2)^{\prime }$ is Fredholm. □

The proof of lemma 1.2 is based on the parameter-ellipticity of equations (1) and (2), which leads to the parameter-ellipticity of the operators F_n and F. The main terms of their symbols are canceled when the difference F_n − F is taken, but the lower order terms still inherit some properties of parameter-ellipticity. So, the proof of lemma 1.2 relies essentially on evaluating the first few terms of the parameter-elliptic operators F_n and F.

Recall that parameter-ellipticity means that the operator of multiplication by $\sqrt{\lambda }$ has the same order as differentiation. More precisely, the full symbol of a parameter-elliptic p.d.o. is an asymptotic series of terms, which are homogeneous in (|ξ|, k), where $k=\sqrt{|\lambda |}$, with the main term (principal symbol) not vanishing when |ξ|² + k² ≠ 0. Thus, we will calculate the first three terms f_j, j = 0, 1, 2, of the full symbol $\sum _{j=0}^\infty f_j$ of the parameter-elliptic p.d.o. F_n (the terms f_j = f_j(x, ξ, λ) are homogeneous functions of ξ and k of order 1 − j) before the proof of lemma 1.2. We will assume that the transformation matrix from global to local coordinates is orthogonal on the boundary $\partial \mathcal {O}$ (this can always be done in dimensions d ⩽ 3). We will need the following result.

Lemma 4.1. Let λ ∈ Λ, where Λ is a closed sector of the complex plane that does not contain points of the set $\mathcal {N}$ defined in (7). Then, the principal symbol f₀ of the parameter-elliptic p.d.o. F_n has the form

$$\begin{equation} f_0=\sqrt{|\xi |^2-n(x)\lambda }. \end{equation} \tag{ 54 }$$

If n(x) ≠ 1 on $\partial \mathcal {O}$, then f₁(x, ξ, 0) does not depend on n(x). If n(x) ≡ 1 on $\partial \mathcal {O}$, then

$$\begin{equation} f_1=g_1(x,\xi ,\lambda )+\frac{\frac{\partial n}{\partial \nu }\lambda }{4(|\xi |^2-\lambda )}, \end{equation} \tag{ 55 }$$

where g₁(x, ξ, λ) does not depend on n(x). The term f₂ is n-independent when λ = 0.

Remark. Obviously, f₀ ≠ 0 when λ ∈ Λ and |ξ|² + |λ| ≠ 0, i.e., F_n is parameter-elliptic.

Proof. The full symbol ∑f_j can be found absolutely similarly to calculations (33)–(39) of the parameter-independent symbol of F_n. However, now all the expansions in generalized homogeneous terms must include $\sqrt{\lambda }$ with the same weight as the weight of |ξ|. We will use notations $\widetilde{A}_{j,\lambda }$ and E_{j, λ} for operators $\widetilde{A}_j$ and functions E_j, respectively, in order to stress that they are different now. In particular,

$$\begin{equation*} \widetilde{A}_{0,\lambda }=-\frac{{\rm d}^2}{{\rm d}t^2}+|\xi |^2-n(x)\lambda . \end{equation*}$$

From (35) it follows that, for n(x) > 0,

$$\begin{equation*} E_{0,\lambda }={\rm e}^{-t\sqrt{|\xi |^2-n(x)\lambda }}, \quad \lambda \in \Lambda , \quad {\rm Re}\sqrt{|\xi |^2-n(x)\lambda }>0,\quad {\rm when}\ |\xi |^2+|\lambda |\ne 0. \end{equation*}$$

Thus, the principal symbol $f_0=-\frac{\partial }{\partial t}E_{0,\lambda }|_{t=0}$ of the operator F_n is equal to (54).

Similarly, one can evaluate $\widetilde{A}_{1,\lambda }$. It consists of two parts: the contribution from the Laplacian (which is defined in (34)) and the contribution from −λn(x). Namely,

$$\begin{equation*} \widetilde{A}_{1,\lambda }=\widetilde{A}_1-\lambda \widetilde{B}_1,\qquad \widetilde{B}_1=-\frac{\partial n}{\partial \nu }t+\left\langle -i\frac{\partial }{\partial \xi },\nabla _y n\right\rangle . \end{equation*}$$

Our next step is to evaluate E_{1, λ} and the second term $f_1=f_1(x,\xi ,\lambda )=-\frac{\rm d}{{\rm d}t}E_{1,\lambda }|_{t=0}$ of the full symbol of the parameter-elliptic operator F_n. Equation (36) for E_{1, λ} has the form $\widetilde{A}_{0,\lambda }E_{1,\lambda }=-\widetilde{A}_{1,\lambda }E_{0,\lambda }$. The right-hand side here and the operator $\widetilde{A}_{0,\lambda }$ do not depend on n(x) when λ = 0. Thus, E_{1, 0} and f₁(x, ξ, 0) are independent of n(x).

Assume now that n(x) ≡ 1 on $\partial \mathcal {O}$. Then, ∇_yn = 0 and E_{1, λ} can be written as E_{1, λ} = G + H, where G and H are the solutions of the equations:

$$\begin{equation*} \left(-\frac{{\rm d}^2}{{\rm d}t^2}+|\xi |^2-\lambda \right)G=-\widetilde{A}_{1}\,{\rm e}^{-t\sqrt{|\xi |^2-\lambda }}, \qquad \left(-\frac{{\rm d}^2}{{\rm d}t^2}+|\xi |^2-\lambda \right)H=-\lambda \frac{\partial n}{\partial \nu }t\,{\rm e}^{-t\sqrt{|\xi |^2-\lambda }} \end{equation*}$$

that vanish at t = 0 and when t → ∞. Obviously, G and $g_1=-\frac{\rm d}{{\rm d}t}G|_{t=0}$ do not depend on n(x). Solving equation for H, we obtain

$$\begin{equation*} H=-\frac{\lambda }{4}\left(\frac{t^2}{\sqrt{|\xi |^2-\lambda }}+\frac{t}{|\xi |^2-\lambda }\right)\frac{\partial n}{\partial \nu }\, {\rm e}^{-t\sqrt{|\xi |^2-\lambda }} , \end{equation*}$$

and therefore,

$$\begin{equation*} -\frac{\rm d}{{\rm d}t}H|_{t=0}=\frac{\frac{\partial n}{\partial \nu }\lambda }{4(|\xi |^2-\lambda )}. \end{equation*}$$

Thus, (55) is proved. In order to find f₂, we need to solve the equation $\widetilde{A}_{0,\lambda }E_{2,\lambda }=-\widetilde{A}_{1,\lambda }E_{1,\lambda }-\widetilde{A}_{2,\lambda }E_{0,\lambda }$. One can easily check that the right-hand side here and the operator $\widetilde{A}_{0,\lambda }$ do not depend on n(x) when λ = 0. Thus, E_{2, 0} and $f_2(x,\xi ,0)=-\frac{\rm d}{{\rm d}t}E_{2,0}|_{t=0}$ are independent of n(x). □

Proof of lemma 1.2. The following fact will be used below. Let Φ be a parameter-elliptic p.d.o. on $\partial \mathcal {O}$ of order s when λ belongs to a closed sector Λ in the complex λ-plane. Then, [1, theorem 4.4.6, 10] for each $m\in \mathbb {R}$, the operator

$$\begin{equation} \Phi :H^{m,k}(\partial \mathcal {O})\rightarrow H^{m+s,k}(\partial \mathcal {O}),\quad \lambda \in \Lambda ^{\prime }=\Lambda \bigcap \lbrace |\lambda |>1\rbrace , \quad k=\sqrt{|\lambda |}, \end{equation} \tag{ 56 }$$

is bounded, as well as its inverse for sufficiently large |λ|, and their norms can be estimated uniformly in λ. Here, $H^{m,k}(\partial \mathcal {O})$ is the Hilbert space with the norm defined in (13).

Denote by K_n(λ) the p.d.o. on $\partial \mathcal {O}$, which is defined by the following two properties. Its construction uses the same partition of unity on $\partial \mathcal {O}$ that was used to define F_n(λ). The symbol of K_n(λ) in each local chart $V\in \partial \mathcal {O}$ consists of the first two terms of the full symbol of the parameter-elliptic operator F_n(λ). Then, the operator K_n(λ) is also parameter-elliptic of order 1. The difference F_n(λ) − K_n(λ) has order −1, i.e.,

$$\begin{equation} \| (F_n(\lambda )-K_n(\lambda ))f\| _{H^{m+1,k}}\le C\| f\| _{H^{m,k}}, \quad \lambda \in \Lambda ^{\prime }, \quad k=\sqrt{|\lambda |}. \end{equation} \tag{ 57 }$$

Denote by K(λ) the operator K_n(λ) with n ≡ 1. Lemma 4.1 implies that the symbol p(x, ξ, λ) of the operator K_n(λ) − K(λ) in V is given by the expression

$$\begin{eqnarray} &&p(x,\xi ,\lambda )=\sqrt{|\xi |^2-n(x)\lambda }-\sqrt{|\xi |^2-\lambda }+f_1(x,\xi ,\lambda ;n)-f_1(x,\xi ,\lambda ;1)\nonumber \\ &&\hphantom{ p(x,\xi ,\lambda )}=\frac{(1-n(x))\lambda }{\sqrt{|\xi |^2-\lambda }+\sqrt{|\xi |^2-n(x)\lambda }}+[f_1(x,\xi ,\lambda ;n)-f_1(x,\xi ,\lambda ;1)]. \end{eqnarray} \tag{ 58 }$$

Here, f₁ is the function defined in lemma 4.1, but we added one more argument in the notation of this function in order to stress that f₁ depends on n = n(x).

Assume now that n(x) ≠ 1 on $\partial \mathcal {O}$. Function (58) is the sum of two terms, which are generalized homogeneous functions of order 1 and 0, respectively. It is important that function p can be written in the form

$$\begin{equation} p(x,\xi ,\lambda )=\lambda [p_1(x,\xi ,\lambda )+p_2(x,\xi ,\lambda )],\quad \lambda \in \Lambda ^{\prime }, \end{equation} \tag{ 59 }$$

where the functions p₁ and p₂ are smooth when |ξ|² + |λ| ≠ 0, generalized homogeneous of order −1 and −2, respectively, and p₁ ≠ 0 when |ξ|² + |λ| ≠ 0. Indeed, the properties of p₁ are obvious, since the relation (which is obtained by equating the denominator in (58) to zero)

$$\begin{equation} \sqrt{|\xi |^2-\lambda }=-\sqrt{|\xi |^2-n(x)\lambda } \end{equation} \tag{ 60 }$$

implies that λ(1 − n(x)) = 0, i.e., λ = 0. Then, (60) requires ξ = 0. Furthermore,

$$\begin{equation*} f_1(x,\xi ,\lambda ;n)-f_1(x,\xi ,\lambda ;1) \end{equation*}$$

vanishes when λ = 0 since lemma 4.1 implies that f₁(x, ξ, 0; n) does not depend on n. Thus,

$$\begin{equation*} p_2=\lambda ^{-1}[f_1(x,\xi ,\lambda ;n)-f_1(x,\xi ,\lambda ;1)], \quad |\xi |^2+|\lambda |\ne 0, \end{equation*}$$

is smooth. Hence (59) holds.

From (59), it follows that

$$\begin{equation*} K_n(\lambda )-K(\lambda )=\lambda R(\lambda ), \end{equation*}$$

where R is a parameter-elliptic p.d.o. of order −1 when λ ∈ Λ' (its symbol equals λ⁻¹p = p₁ + p₂). Thus, the operator

$$\begin{equation*} |\lambda |^{-1}[K_n(\lambda )-K(\lambda )]:H^{m,k}(\partial \mathcal {O})\rightarrow H^{m+1,k}(\partial \mathcal {O}),\quad \lambda \in \Lambda ^{\prime }, \quad k=\sqrt{|\lambda |}, \end{equation*}$$

is uniformly bounded in λ. This and (57) imply (14). Let us prove (15). We have

$$\begin{eqnarray} &&F_n-F=K_n-K+(F_n-K_n)-(F-K)=R(\lambda )\lbrace \lambda I+R^{-1}(\lambda )T_1\rbrace ,\quad \lambda \in \Lambda ^{\prime },\quad |\lambda |\gg 1, \nonumber\\ \end{eqnarray} \tag{ 61 }$$

where I is the identity operator and the operator T₁ = (F_n − K_n) − (F − K) has order −1. It remains to note that

$$\begin{equation*} \| R^{-1}(\lambda )f\| _{H^{m,k}}\le C\| f\| _{H^{m+1,k}}, \quad \lambda \in \Lambda ^{\prime }, \quad k=\sqrt{|\lambda |}\rightarrow \infty , \end{equation*}$$

i.e., R⁻¹(λ)T₁ is a bounded operator in H^{m, k}, and therefore (61) leads to (15).

Assume now that $n(x)\equiv 1, \frac{\partial n}{\partial \nu }\ne 1$ on $\partial \mathcal {O}$. Then, we denote by K_n the p.d.o., whose symbol in each chart $V\in \partial \mathcal {O}$ consists of the first three terms of the full symbol of the parameter-elliptic operator F_n(λ). Then, the operator T₂ = (F_n − K_n) − (F − K) has order −2 and the operator K_n − K has the form K_n − K = λR(λ), where R(λ) is a parameter-elliptic p.d.o. of order −2 (its principal symbol is $\frac{\frac{\partial n}{\partial \nu }}{4(|\xi |^2-\lambda )}$, and the remaining part of the symbol is λ⁻¹[f₂(x, ξ, λ; n) − f₂(x, ξ, λ; 1)]). Now, the second part of the statement of the theorem follows easily from the representation:

$$\begin{equation*} F_n-F=K_n-K+(F_n-K_n)-(F-K)=R(\lambda )\lbrace \lambda I+R^{-1}(\lambda )T_2\rbrace ,\quad \lambda \in \Lambda ^{\prime }. \end{equation*}$$

 □

We will continue to assume infinite differentiability of $\partial \mathcal {O}$ and n although a finite smoothness is enough for all the results of this paper. We will call (u, v) a weak interior transmission eigenfunction if it satisfies relations (1), (4), (3) with weakened assumptions on smoothness of u and v. One option is to assume that

$$\begin{equation*} u\in L_2(\mathcal {O}), v\in L_2(\mathcal {O}\backslash \mathcal {V}), u-v\in H^2(\mathcal {O}\backslash \mathcal {V}). \end{equation*}$$

The last inclusion allows one to define the Dirichlet and Neumann boundary values for u − v on $\partial \mathcal {O}$ and therefore there are no difficulties in making the boundary condition (3) meaningful. In fact, one does not need to assume that $u-v\in H^2(\mathcal {O})$. Of course, one cannot define the trace at the boundary for arbitrary square integrable functions. However, the traces $u,v\in H^{-1/2}(\partial \mathcal {O}), u^{\prime }_\nu ,v^{\prime }_\nu \in H^{-3/2}(\partial \mathcal {O})$ are well defined [19] for functions $u,v\in L_2(\mathcal {O})\times L_2(\mathcal {O}\backslash \mathcal {V})$ if the latter functions satisfy equations (1) and (4) (or other elliptic equations). The traces are understood in this case as limits of corresponding traces for smooth approximations of the solutions u, v. Thus, relations (3) for weak interior transmission eigenfunctions $(u,v)\in L_2(\mathcal {O})\times L_2(\mathcal {O}\backslash \mathcal {V})$ are understood as equalities in the spaces $H^{-1/2}(\partial \mathcal {O}), H^{-3/2}(\partial \mathcal {O})$, respectively.

Theorem 5.1. Let condition (5) or (6) hold. Then, $u,v\in H^2(\mathcal {O})$ for any weak solutions $(u,v)\in L_2(\mathcal {O})\times L_2(\mathcal {O}\backslash \mathcal {V})$ of the equations (1), (4), (3), i.e., the set of weak ITE-s coincides with the set of ITE-s in the strong sense defined earlier.

Proof. First assume that a weak ITE λ = λ₀ is neither a pole of F(λ) nor a pole of F_n(λ). Let (u, v) ∈ L₂ × L₂ be the corresponding weak interior transmission eigenfunction. If $\varphi :=u=v\in H^{-1/2}(\partial \mathcal {O})$ is the value of u = v at the boundary, then from (3) it follows that [F(λ₀) − F_n(λ₀)]φ = 0, i.e., φ belongs to the kernel of the elliptic (see lemma 1.1) p.d.o. on $\partial \mathcal {O}$, and therefore $\varphi \in C^\infty (\partial \mathcal {O})$. Then, L₂-solutions u, v of the Dirichlet problems for equations (1) and (4) with the Dirichlet data $\varphi \in C^\infty (\partial \mathcal {O})$ are infinitely smooth (see [19]).

Assume now that a weak ITE λ = λ₀ is a pole of F(λ) or F_n(λ) (or both of these operators). Then, from (3) it follows that $\varphi =u|_{\partial \mathcal {O}}=v|_{\partial \mathcal {O}}$ belongs to the kernel of both the residue of [F(λ) − F_n(λ)] at the pole λ = λ₀ and the principal part of [F(λ) − F_n(λ)] at the pole λ = λ₀. The latter property again implies that $\varphi \in C^\infty (\partial \mathcal {O})$ since the principal part is also an elliptic p.d.o. on $\partial \mathcal {O}$, see lemma 1.1. Hence, u, v ∈ C^∞ in all cases. □

We are very grateful to the referees for their remarks, which improved the paper significantly and allowed us to avoid some mistakes. The work of EL was supported by FEDER funds through COMPETE–Operational Programme Factors of Competitiveness (Programa Operacional Factores de Competitividade) and by Portuguese funds through the Center for Research and Development in Mathematics and Applications (University of Aveiro) and the Portuguese Foundation for Science and Technology (FCT–Fundção para a Ciência e a Tecnologia), within project PEst-C/MAT/UI4106/2011 with COMPETE number FCOMP-01-0124-FEDER-022690, and by the FCT research project PTDC/MAT/113470/2009. The work of BV was partially supported by the NSF grant DMS-1008132.